I’d like to try posting the logs of nixpkgs-update/r-ryantm in this thread. The logs show which packages nixpkgs-update is failing to automatically update, and reviewing them could indicate where manual intervention is needed.
I don’t think there is a general solution why a package failed to build.
I wonder how we can visualize this information better.
One approach could be to create branches for the each package update,
where the commit messages pings the maintainer and includes the last 500th lines of build log or so.
Then those maintainer could pull the branch and maybe fix the build.
For many packages I can already tell from the error log what went wrong.
I’m excited you would love to help! There are a bunch of different ways the nixpkgs-update can fail; some of them require manual intervention, and some of them require improvements to nixpkgs-update. I’ll try to cover some of the failures that need manual intervention.
nixpkgs-update log error messages that require manual intervention
“nix build failed”
This error means nixpkgs-update successfully updated the src fixed-output derivation to the latest version, but nix-build failed. Either the package failed to build before, or the update caused it to fail. This could mean one of its dependencies is broken, a new dependency needs to be added, or something about the build process needs to be updated. The nixpkgs-update log contains the last few lines of the build output, which could be useful in seeing what went wrong, but you probably should try manually updating the package and running nix-build to see the full context.
“Could not prefetch new version URL”
Either the information from Repology.org is wrong, or upstream changed how the src is fetched. Try to find out where the new src version is obtained from and see if it builds.
“More than one fetcher in”
It refuses to update any derivation in a file that contains more than one fetcher derivation, because it isn’t smart enough to update these. These packages need to be updated manually. If you want nixpkgs-update to update these derivations in the future, try to break out the derivations into separate files, so there is only one fetcher in each file.
“nix-env -q failed to find package name with old version”
It can’t find the derivation file for the package, so it needs to be updated manually until nixpkgs-update is smart enough to find the derivation file.
Conclusion
There are more errors than this, but these ones should keep a bunch of people busy! If you have questions about other error messages, please ask. I’d be glad to help explain more.
Thanks for the precise information.
Will try to give a hand. For example, in the below case, r-ryantm just needs to be told that the 2 in davfs2 is not linked to the version number. There could be a davfs2 v0.4 as well as a v3.6, right ?
2018-11-10T11:21:29 davfs2 1.5.3 -> 1.5.4
2018-11-10T11:21:37 Version in attr path davfs2 not compatible with 1.5.4
2018-11-10T11:21:37 FAIL
I am not that much into fixing packages by hand, but I would be glad to try to make r-ryantm smarter ;-).
I also like the idea of fixing download urls seems like a good idea, because it is a medium-term fix.
We could add a list of packages to ignore the version attrpath compatibilty check. The general solution to this is to make all package names conform to an unambiguous convention like I propose in
There is definitely too much in these logs. I wrote a quick script to split them at split.sh · GitHub
There is a lot of work to do with prefetch not working (761 cases), there is a weird number of successes (217) and a expected number of failed builds (362).
Great, I was able to run your script inside nix-shell -p parallel by renaming the log file to r-ryantm.
Please note I slightly changed the log format toward the end of this log. I put the failure message on the same line as FAIL. I haven’t spent the time to think of a good format for the log messages, if you have some ideas I’d be happy to hear them.
@ryantm Are more recent logs available anywhere? I came to have a look to see if I could help at all this afternoon but couldn’t see where the logs from recent runs were.
@mpickering Thanks for helping out! The logs are on my computer :). I haven’t been uploading them lately because the current code hasn’t been able to make it reliably through a complete run. That said, old logs should still be a source of work that needs doing, because most of the errors will only be fixed by human intervention (and it should be easy to see if someone already fixed it).
@ryantm: maybe nixpkgs-update could the logs as well either in a web server with directory listing or something else simple like git with lfs-enabled. What would also help for parsing, if packages are seperated by two newlines, and compile ouput would never contain more then one newline.
The issue with dub is not that it has multiple derivations. The issue is that the main derivation does not have a src attribute, which it checks for to make sure the changes it makes to the file actually update the sources for the main derivation. If you moved the src definition to the let block, and inherited it into the main derivation without it being used for anything, it would probably work. But it’s not so great to have the updater imposing something weird like that, so we should try to think of a better way.
I created an alias in Repology from solc to solidity (because that’s what most distributions use). nixpkgs uses the solc name - will nixpkgs-update figure this out and detect the new version?