Automated tests could replace a lot of this. For example nixos tests catch a lot of problems but are also way underused.
edit: there are 478 nixos tests total, for 50000+ packages!
4 Likes
Note that contributors (as of recently) have been encouraged not to use NixOS tests for regular packages, since they are relatively expensive (fire up a VM, etc.). The passthru.tests attribute should be used instead.
Unfortunately, we also only have ~215 passthru tests.
5 Likes
seems like test coverage is something that could be improved with funding, it’s relatively simple and something noone wants to do, so put bounties on tasks like this, and some students will get it done relatively quickly
2 Likes
I haven’t used Ubuntu in a number of years, but they had the idea of a primary repository that was supposed to be “perfect”, and then “universe” and “multiverse” repositories (or whatever they were called) that were of varying degrees of quality, but usually decent enough. It wasn’t AUR or NUR, it was still Ubuntu community maintained, but with at least some standards (IIRC)
I wonder if introducing the idea of support tiers for packages would be helpful at all. Packages are still maintained in our monorepo, but we can apply more liberal automatic merge logic to them.
Just a random thought, maybe I’m off the mark entirely.
8 Likes
Part of this is, I think, documentation. The only mention in the nixpkgs manual, in the meta attribute section, is thin on how you’d write a test from scratch or what would be meaningful. Most of the examples in nixpkgs are simply selecting tests from nixosTests, so it takes a bit of hunting to get a sense of what patterns are “normal”, let alone best-practice.
I recall this coming up in the course of Improve a pkgs' tests for nixpkgs-update, but it doesn’t look like that produced anything yet (I think mostly because it was focused on the bot, and broke down around where the docs should go?)
No objection to bounties, but I also wonder if test-writing would feel more meaningful if we had:
- a meta attribute for marking packages that are well-tested enough for updates to be auto-merged if they build and passthru passes
- a standard for deciding when packages are well-tested enough to grant this attribute
- a process for deciding whether a package is sufficiently tested to meet the standard?
In the short-er term, I imagine test-writing would be more intellectually rewarding (than update/review work) if the result was a significant improvement in the maintainability of a package (and the knowledge that, at scale, you’re helping rebalance what human maintainers spend time on?) Maybe @jonringer has a sense of whether that feels less Sisyphean?
Additional levers could be applied…
- nudging people who use under-tested packages to submit real test cases
- requiring them on new packages
5 Likes
Do you expect that an updated version of, say, [RFC] nixpkgs/tests: create nixos-independent PoC tests by 7c6f434c · Pull Request #36842 · NixOS/nixpkgs · GitHub will not get replies like [RFC] nixpkgs/tests: create nixos-independent PoC tests by 7c6f434c · Pull Request #36842 · NixOS/nixpkgs · GitHub this time around?
Of course https://github.com/NixOS/nix/pull/3600 or its updated version Merge master into #3600 by Ericson2314 · Pull Request #4155 · NixOS/nix · GitHub could also work for making a cheap version of NixOS tests…
A question was raised, with everyone who touched that package or filed issues or PRs about it in the last 3 years mentioned, and in 2 weeks no non-retracted objections are outstanding?
(Basically, if there are too many caring about the package to get to an agreement, maybe it is not a niche package and it has a chance of substantial reviews in the first place)
I said what I said above, because you don’t need to be an organizational expert to see with your own eyes, that necesarily the individually percieved owerhead on smaller units is far less, and so is the mooting effect of that overhead.
And if there is an overhead that really gets into a contributor’s way, they can at least perceive a path to realistically get rid of it.
Non of that is possible in nixpkgs for more than 95% of contributors (at least) — and within reasonable time.
I’ve tried some things and my success rate was far less than 50%. So I decided not to bother anymore and redirect my energy (sucess induced endorphine rush while working on devos is close to 100%). That’s what’s happening.
I’m not saying that I would be a notable loss to the community, by any means. It’s just an ordinary perspective of an ordinary contributor.
→ Peter Drucker & Stafford Beer read’em up!
Personally I was driven away by the attitude that we don’t need tests or they are just an unnecessary burden for large features to go in and the fact that apparently almost nobody cares about documenting their changes properly in the commit messages.
So I question: Do you actually want tests in nixpkgs or are those that want tests just those that are vocal about it?
Having tests goes both ways. It isn’t just having them it also means maintaining them. If someone comes by an drops a new package with a large test suite that is great BUT do you want to maintain that without being an expert on say a random data science package?
10 Likes
That sounds more like:
I imagine a standard having to more to say about what should be tested (probably actually a few different standards for different types of package) and ~how (technically, level of detail, etc.)
1 Like
You see, were there smaller fedarated units that make up the Fedarate Republic of Nixpkgs, you’d had had a light time introducing proper testing rules in one subunit that was ripe for it.
And good ideas usually spread.
As a german, I know what we owe to federalism.
Ceterum censeo:
→ Peter Drucker & Stafford Beer reed’em up!
M R via NixOS Discourse nixos1@discoursemail.com writes:
Do you expect that an updated version of, say, [RFC] nixpkgs/tests: create nixos-independent PoC tests by 7c6f434c · Pull Request #36842 · NixOS/nixpkgs · GitHub will not get replies like [RFC] nixpkgs/tests: create nixos-independent PoC tests by 7c6f434c · Pull Request #36842 · NixOS/nixpkgs · GitHub this time around?
I think (hope?) that if it were presented as an RFC as per our current
process, maybe linking to this PR as implementation work, then the
community as a whole would have more impact than a few individual
voices, and it would have more chances of getting accepted.
1 Like
Effectively procuring proper direction abiding by the highest (and most effective) inter-human relationship standards for a group of more than 7 people is almost impossible.
That’s what management litterature has to teach us.
Member States havn’t to be as small as that, but usually there is tacit delegation to trusted individuals happening in an open source community context, so homogenous groups of trusted individuals that pull together on the same side of the rope, probably would be a sufficient (social) splitting criterion for the Federate Republic of Nixpkgs.
Ceterum censeo:
→ Peter Drucker & Stafford Beer reed’em up!
Also, have you guys considered the fact that a Federate Republic of Nixpkgs creates a few naturally aspirable vacancies that currently do not exist?
Those new vacancies are meaningful and rewarding.
You can argue that people need paychecks to solve the problem as much as you like, but people in open source are not intrinsically motivated by monetary retribution. At least not if you want great outcomes in return.
Ceterum censeo:
→ Peter Drucker & Stafford Beer reed’em up!
Andreas Rammhold via NixOS Discourse nixos1@discoursemail.com writes:
Having tests goes both ways. It isn’t just having them it also means maintaining them. If someone comes by an drops a new package with a large test suite that is great BUT do you want to maintain that without being an expert on say a random data science package?
Is it actually a problem? IMO tests are just a way to make building a
package actually OK only when the package works.
IOW, if said random data science package is not important, then we can
just break the tests with unrelated changes without caring much — and
then the next person who’ll want to use the package will have to fix the
tests. Basically, the same thing as we do for packages that stop
building.
OTOH, if the random data science package is important… then missing the
fact that it stopped working due to the unrelated changes was bad and
something we would want to avoid. Making this visible before the
unrelated changes land is then a good thing, and can allow us to poke
the maintainer of that important package before landing, to fix the
thing.
TL;DR: I don’t think there’s any additional maintenance burden for
having more tests that would get imposed on people who don’t already
maintain the tested packages, at least so long as the tests don’t
significantly slow down automation by just being slow.
Well, I guess we are falling back on how to
draw the rest of the fucking owl
That doesn’t move us one step further in the issue of “contributor retention”.
1 Like
I think this is more in regards to https://hydra.nixos.org/eval/1662102?filter=nixos.tests&compare=1661891&full=#tabs-still-fail , in which there’s always an ample supply of failing tests. More comprehensive the test, more likely it is to break.
Eventually, the “bill comes due” for ZHF, in which there’s an effort to fix the builds and tests.
2 Likes
I think this is more in regards to Hydra - Evaluation 1662102 of jobset nixos:trunk-combined , in which there’s always an ample supply of failing tests. More comprehensive the test, more likely it is to break.
Which only gets noticed when a package gets updated and the tests subsequently gets looked at, or if the test is part of the tested set. I last touched babeld in 2020/09 and now it broke in 2020/12 and I’m not sure how I ever could have noticed without polling for it.
4 Likes
Many of the most famous and successful open source projects were developed or maintained long-term by paid staff. The Linux kernel is largely developed by paid developers (Intel, Red Hat, etc.). Red Hat employs a large number of the glibc, gcc, X.org, Wayland, and GNOME developers. Rust started in Mozilla (even though it became independent since then). LLVM was developed at University of Illinois at Urbana–Champaign and then was further developed by Apple (and now many companies). LibreOffice started as StarOffice by Star Division, was then bought by Sun and open sourced.
This ‘work for free’ attitude is a pipe dream for companies who only want to benefit from open source and not contribute back. But in the end everyone needs to eat, rent/buy a house, travel, see the world, go out. There is nothing wrong with open source as a hobby, but there is also nothing wrong at all with receiving compensation as one of the motivators. I’d rather see good compensation for developing open source, so that smart people write great open source software, than have them write proprietary software.
Back on-topic. I think there is a lot of less interesting work, were I think it would be great if we could pay people to do it. There was a fundraiser to get better error messages in Nix – not the most exciting work to do, but it benefits the community hugely. macOS support is often lagging behind, because we do not have a lot of maintainers on macOS, so it’s great that @domenkozar and others raised money, so that someone can be paid now to improve macOS support. There are some other areas where I think funding someone for several hours per week would be very beneficial (e.g. triaging CVEs).
16 Likes
Nothing wrong with getting paid, still you unfortunately absolutely misread my point, which is somewhat informed by HR research in compensation strategies.
Empowerment and meaningfulness score much higher on the Maslow Pyramid and are much, much more important motivating factors for an open source context.
NixOS/nixpkgs absolutly abysmally fails at this, especially at empowerment.
Breaking stuff up and creating enticing vacancies in the Federated Republic of Nixpkgs, is just a “simple” “solution”. Can’t help but to stress that again, against the odds of “drawing the fucking owl” around here.
Note, that my baseline assumption here is that people lingering around open source projects neither starve nor lack a place to sleep (referring to the lower echelons of the Maslow Pyramid).
Ceterum censeo:
→ Peter Drucker & Stafford Beer reed’em up!