Improve a pkgs' tests for nixpkgs-update

From nixpkgs-update, I got pinged:

  • The tests defined in passthru.tests , if any, passed
  • 0 of 0 passed binary check by having a zero exit code.
  • 0 of 0 passed binary check by having the new version present in output.

How do I write these tests here?

@ryantm Maybe could a TL;DR answer to this cuestion be made discoverable from the bot’s comments? That would be awesome.

Also when tested, what am I supposed to respond to the bot? LGTM?


EDIT:

Despite all the pointers in here, I’d still need help on how to correctly interpret and transact the checks section into example package tests.


1 Like

https://nixos.org/manual/nixpkgs/stable/#sec-standard-meta-attribute documents it. Basically it’s a list of derivations that have to build for the tests to pass.

It sounds like a good idea to link to general update bot docs.

The best response is a GitHub review that is an approval with a comment about what you tested

3 Likes

Thank you!

Two follow-up question:

  • How does updateWalker relate to nixpkgs-update?
  • What does nixpkgs-review in addition to nixpkgs-update? (In this example it appeared to me they as they did the same thing twice)

Also for the record, from this entry point into the topic, the concept of passthru is neither clear nor discoverable.

So, as a person wanting to write tests in response to the update bot, I’d have to jump over another hurdle and figure that out before. But that’s probably a minor inconvenience.

1 Like

nixpkgs-review is a tool that builds all the packages in nixpkgs that are affected by a given PR or commit, then it drops you into a shell where you can inspect the build results, and it generates a report about what built and what didn’t.

nixpkgs-update is mainly the software that runs the r-ryantm bot, which collects version update information from external sources (Repology, GitHub releases, PyPI) and uses that to attempt to update packages one at a time. If it succeeds at building the updated package, it runs some checks on it (including nixpkgs-review) and makes a PR. nixpkgs-update can be used as a cli to generate your own package updates and PRs, but it is not very polished for that purpose.

More broadly/philosophically nixpkgs-update is my initiative to keep nixpkgs as up to date as possible, so any work that helps get things more up to date, including improving general PR throughput, is fair game.

Nixpkgs-update doesn’t do anything with updateWalker. Here is a description of updateWalker: https://github.com/NixOS/nixpkgs/blob/92a047a6c4d46a222e9c323ea85882d0a7a13af8/pkgs/build-support/upstream-updater/update-walker.txt

3 Likes

@ryantm I’d suggest we initiate a .github/R-RYANTM-HELP.md similar to .github/STALE-BOT-HELP.md for guiding people around. What do you think?

This R-RYANTM-HELP.md file should be in the nixpkgs-update repo, where it can be kept up to date. r-ryantm is actually an unprivileged user of the NixOS repo, so it doesn’t make sense for there to be “official” information about it.

I perceive it as a current shortcoming that there is no official documentation of this de-facto standard. I think legitimacy (to make it “official” — under .github) comes out of the value added and the integral part that r-ryantm plays in the ecosystem, if that should be a concern.

I agree on the updating aspect.

All in all, from a end user UX, though, I would want those kind of helps be (officially) discoverable directly from grep NixOs/nixpkgs or — judging by adjacent on-topic content — directly within .github/.

What is the main objection to make it “official”?

Besides automated tests with passthru.tests, I believe there is also installCheck?

1 Like

there’s a checkPhase and an installCheckPhase and they correspond to normal make check and make installCheck. The difference is that those phases have to succeed or else it will fail the build.

passthru.tests allows you to run tests which interacts with many packages (e.g. nextcloud + postgres) or allows you to run downstream dependencies (e.g. python3Packages.dash + python3Packages.dash-XXX)

For packages that have a related module, it’s common to create a nixosTests.XXXX which will validate that module still works as well. These can also be added to passthru.tests

3 Likes

So I’m reading:

answered by @jonringer

I assume passthru.updateScript neither is an input for nixpkgs-update.

And here is @garbas 's (it’s author’s) stance from 2018 https://github.com/ryantm/nixpkgs-update/issues/12#issuecomment-373953813 — still valid?

See also:

nixpkgs-update is passthru.updateScript aware, and if a package contains one, then it will use the update script to attempt to update it.

Example PR: https://github.com/NixOS/nixpkgs/pull/100772

you can see at the top it states:

Automatic update generated by nixpkgs-update tools. This update was made based on information from passthru.updateScript.

If a package does have an update script, it can be invoked with:

nix-shell maintainers/scripts/update.nix --argstr package <pkg-name>

Essentially this works by “dropping” you into an impure shell, then executing the script in a shellHook. so the changes will affect your local repo.

1 Like

So I’m reading:

Which I read as every language has it’s “own” test tooling. Would they all be triggered by nixpkgs-update without further a due?

(asking, so I can encode that knowledge into the PR)

python is an awkard ecosystem, so there’s additional toolling around handling common scenarios within python.

It’s not standard for all languages to have such tooling, especially when a language has a standard way to test all packages (python does not, e.g. unittest vs pytest vs nose)

setuptools-check-hook.sh is semi-deprecated because the underlying test setuptools command is deprecated, but kept around for legacy reasons.

3 Likes

I finally documented it today!

2 Likes

Thanks! And happy birthday!