Improve the JDK infrastructure on nixpkgs

Development
3

feel free to tag me for reviews - I am no Java expert and don’t use it day-to-day but we need better support in NixOS for it IMO I’d hope @fzakaria would be involved. I know he’s done some work with Java and NixOS

2 Likes
4

I bet some of this could be simplified with a separate repo overlay people can adopt ?

I am lately into radical simplification.
Part of the challenge with Nixpkgs (to me) is it accepts too much.

I saw recently a PR to cleanup some JDK stuff which I looked at it and commented I’d be in favor of going further and just consolidating on OpenJKD.
(Anything else should be an overlay outside nixpkgs)

4 Likes
5

I totally endorse this effort, and would like to allocate some of my time to it (if that would help even a little)!

I think that the problem behind poor support for JVM languages in Nixpkgs is not only due to technical difficulties, but also because of a certain stigma that the whole Java-adjacent ecosystem has garnered over the last decade or so. I firmly believe that the JVM is a fine piece of software that still has much to give, if used correctly, together with the astonishing amount of well-written and feature-complete libraries that build upon the JDK (for which suitable replacements do not exist at all, sometimes).

I can totally see the reasoning behind this. Maybe we should just maintain a solid, accessible and trivial building infrastructure within Nixpkgs, and then delegate something like a full-fledged Maven builder to external overlays/flakes? If that is the case, we should strive for maximum “pluggability” of the Nixpkgs-side of things, I guess…

1 Like
6

Maybe we should track the above in GitHub? Tackle the first few things and start to gather folks the might be interested in the effort?

7

I too support this effort, and would like to be pinged for reviews.

It’s important to note also, that (in my experience) many jdk depending packages are not source based, and are just a makeWrapper packages that wrap a .jar file downloaded from a GitHub release page. That’s a problem because we don’t have any way to tell if such a packages break when we update JDK, and that’s why people are afraid to support merging major version updates of JDK packages, which a red 10.rebuild-linux: PR label.

I think we should add a --version test in an installCheckPhase or something like that, for all of those .jar based packages. This would help nixpkgs-review catch breakages when we perform a major JDK update. It may very well be, that many of these packages haven’t been updated in years even by their upstream developers (usually games in my experience), and hence we should not make any effort to fix these.

A quick git grep showed me there are 76 such packages:

git grep -l -E 'url = .*.jar";' pkgs/**/default.nix
3 Likes
8

Thanks a lot for all your responses! There are a lot of TODOs, but I have a very positive feeling about the future of JDK on nixpkgs.

I started to create issues to start some more technical discussion and track progress: [Umbrella issue] Improve JDK infrastucture · Issue #313216 · NixOS/nixpkgs · GitHub

So far I linked to 3 PRs on the umbrella issue, it’d be great if you could take a look to those.

3 Likes
9

Thanks for kicking off this effort @jlesquembre.

I am using Java daily on nixos and darwin targets, and so far I found the support to be really good.

I don’t have commit access, but I have been looking for meaningful ways to contribute back to the community. Happy to help with code reviews and testing (especially on aarch64 darwin) if needed. I’d also be happy to work on PRs if I manage to get up to speed.

1 Like
10

As the author of the big bump PR that seems to have kicked all of this off, I’m glad to see people excited to discuss it! As mentioned, please take a look at the PRs linked in the umbrella issue! The big bump, which should address a lot of the outdated packages, has been ready to merge for quite a while now, and I hope it gets merged sometime in the next 3 months.

Once I get a bit more time, I’ll try and put down my thoughts on the various sub-issues linked in the umbrella issue.

1 Like
11

I think we should add a --version test in an installCheckPhase or something like that

testers.testVersion?

There is no updateScript for JDK derivations.

I do not believe this is “possible”, since not every Java program builds on all JDK.

Also, we have a much more serious problem in my modest opinion: Java bootstrap depends on open source abandonware.
I do not know how much Nixpkgs care about bootstrappability, but I believe this is far from a serious priority.

12

I am a maintainer of two such packages: jquake and protege-distribution.
While the first one is a proprietary application, I packaged the second like that because, at the time, I couldn’t figure out a way of building complex Maven applications with an acceptable level of maintenance burden on the Nixpkgs side of things.

If we manage to get a good solution for Maven, then maybe I could deprecate that package in favor of a source-built one.

13

As the maintainer of maptool, I would also be very happy to move to a source-built package if there was a better story for managing Gradle dependencies.

14

The difference, is that installCheckPhase makes the build fail if --version | grep ${version} fails, whereas testers.testVersion is not being run by nixpkgs-review.

15

You can try out our (somewhat new) maven.buildMavenPackage.

I’d also encourage you to add a some kind of an installCheckPhase for the proprietary package, so we’ll be able to know via nixpkgs-review if it gets broken without running it ourselves.

1 Like
16

whereas testers.testVersion is not being run by nixpkgs-review

Is this a bug or a feature of nixpkgs-review?

2 Likes
17

looks like a missing feature Add flag to build package `passthru.tests` by B4dM4n · Pull Request #397 · Mic92/nixpkgs-review · GitHub

1 Like
18

Even with that PR, or something alike merged, people won’t get used to add the --tests flag to their nixpkgs-review runs, so you might still be missing breakages. I don’t understand this insistence on passthru.tests.testVersion where an installCheckPhase is a much stronger and simpler compliance that a package can fulfill.

1 Like
19

The Nixpkgs manual says:

Tests that are part of the source package are often executed in the installCheckPhase.

Prefer passthru.tests for tests that are introduced in nixpkgs because:

  • passthru.tests tests the ‘real’ package, independently from the environment in which it was built
  • we can run passthru.tests independently
  • installCheckPhase adds overhead to each build

Should the manual be changed? Or nixpkgs-review? Or people’s habits for how and when they use nixpkgs-review versus other commands?

1 Like
20

Thanks for quoting Nixpkgs’ manual :slight_smile: . This was discussed a bit at #119731 but people gave their attention in the discussions there on topics other then this. Someone there did raise my opinion, but nobody commented on this opinion…

I noted myself to discuss this at Nixpkgs at some point when I’ll have time.

2 Likes
21

If the problem is human failure, then it can and should be fixed by automation.
The CI should add those flags, then.

2 Likes
22

An aside from folks sorta getting their feet wet (the toes went first) with Java - what JDK is ideal for nixpkgs work? Does it even matter? Please feel free to DM me as to not derail overall thread. Many thanks and this feels great. @doronbehar hope you are well!