Improve the JDK infrastructure on nixpkgs

The Problem

The JDK needs better support in nixpkgs. The JDK ecosystem is large and critical, and improved support would benefit not only the Java ecosystem but also many others, such as Kotlin, Scala, Clojure, and Groovy.

I am grateful for the past efforts of contributors to the JDK infrastructure. But my impression is that many, if not all, have moved on to other projects, leaving the JDK ecosystem with very limited support. I also believe that currently, no one with commit access is particularly interested in the JDK. Some of the issues I consider urgent include:

  • JDK versions are not updated automatically. There is no updateScript for JDK derivations.
  • JDK versions are updated slowly. For example, the current JDK LTS is 21. We are on version 21.0.0, while the latest release is 21.0.3. Nixpkgs unstable is three versions behind.
  • JDK-related PRs, even minor ones, take a long time to be reviewed and merged. For instance, I submitted a PR to update the default JDK from version 19 to version 21. Version 19 was already end-of-life and marked as insecure at the time, but it took approximately three months to be merged.
  • We have too many JDK flavors in nixpkgs. Some of these are likely unnecessary and could be removed. For example, AdoptOpenJDK is a deprecated project replaced by Adoptium (which is also available in nixpkgs).

There are many more areas for improvement, but in my opinion, we should first address the points mentioned above.

We need to put more effort into the JDK infrastructure and seek the help of contributors with commit access to unblock existing PRs and incentivize new ones.

Next steps and call to action

First and foremost, I’d like to identify anyone interested in helping improve the JDK infrastructure on nixpkgs. My goal is to achieve incremental improvements and get changes merged efficiently. To achieve this, even just getting some reviewers and testers for PRs would be extremely helpful. We can discuss the technical details in a separate thread.

Who is interested in contributing to a better JDK infrastructure on nixpkgs? Is anyone with commit access to nixpkgs interested in lending a hand with this effort?

On my end, I’ll continue trying to improve the JDK ecosystem on nixpkgs. For now, I’m putting together a list of ideas for what to do next, and I’ll share it once I’ve cleaned it up a bit.

16 Likes

I’m interested because I regularly rely on the JDK ecosystem a lot, and I do have commit access - but also I’m already drowning in projects so I don’t plan on adding any new significant ones :wink: .

As a positive aside, practically speaking I’ve been mostly very happy with the Java support in nixpkgs.

I think part of the stagnation might be caused by a fear of breaking things. I think it would be reasonable to explicitly say, since the JVM ecosystem is low on maintainers, we should move a bit faster on the shared infrastructure, and if that breaks downstream derivations, that might be an incentive to either remove those derivations from nixpkgs, or encourage their maintainers to help out with the shared infrastructure more, or encourage more users to help out with maintenance.

Awesome!

I’m a little concerned we can generate ideas much faster than we can implement them, but good to collect them somewhere. Maybe we should turn them into GitHub issues with topic:java and kind:enhancement tags when they’ve been cleaned up, so we can discuss them there?

2 Likes

feel free to tag me for reviews - I am no Java expert and don’t use it day-to-day but we need better support in NixOS for it IMO I’d hope @fzakaria would be involved. I know he’s done some work with Java and NixOS

2 Likes

I bet some of this could be simplified with a separate repo overlay people can adopt ?

I am lately into radical simplification.
Part of the challenge with Nixpkgs (to me) is it accepts too much.

I saw recently a PR to cleanup some JDK stuff which I looked at it and commented I’d be in favor of going further and just consolidating on OpenJKD.
(Anything else should be an overlay outside nixpkgs)

4 Likes

I totally endorse this effort, and would like to allocate some of my time to it (if that would help even a little)!

I think that the problem behind poor support for JVM languages in Nixpkgs is not only due to technical difficulties, but also because of a certain stigma that the whole Java-adjacent ecosystem has garnered over the last decade or so. I firmly believe that the JVM is a fine piece of software that still has much to give, if used correctly, together with the astonishing amount of well-written and feature-complete libraries that build upon the JDK (for which suitable replacements do not exist at all, sometimes).

I can totally see the reasoning behind this. Maybe we should just maintain a solid, accessible and trivial building infrastructure within Nixpkgs, and then delegate something like a full-fledged Maven builder to external overlays/flakes? If that is the case, we should strive for maximum “pluggability” of the Nixpkgs-side of things, I guess…

1 Like

Maybe we should track the above in GitHub? Tackle the first few things and start to gather folks the might be interested in the effort?

I too support this effort, and would like to be pinged for reviews.

It’s important to note also, that (in my experience) many jdk depending packages are not source based, and are just a makeWrapper packages that wrap a .jar file downloaded from a GitHub release page. That’s a problem because we don’t have any way to tell if such a packages break when we update JDK, and that’s why people are afraid to support merging major version updates of JDK packages, which a red 10.rebuild-linux: PR label.

I think we should add a --version test in an installCheckPhase or something like that, for all of those .jar based packages. This would help nixpkgs-review catch breakages when we perform a major JDK update. It may very well be, that many of these packages haven’t been updated in years even by their upstream developers (usually games in my experience), and hence we should not make any effort to fix these.

A quick git grep showed me there are 76 such packages:

git grep -l -E 'url = .*.jar";' pkgs/**/default.nix
3 Likes

Thanks a lot for all your responses! There are a lot of TODOs, but I have a very positive feeling about the future of JDK on nixpkgs.

I started to create issues to start some more technical discussion and track progress: [Umbrella issue] Improve JDK infrastucture · Issue #313216 · NixOS/nixpkgs · GitHub

So far I linked to 3 PRs on the umbrella issue, it’d be great if you could take a look to those.

3 Likes

Thanks for kicking off this effort @jlesquembre.

I am using Java daily on nixos and darwin targets, and so far I found the support to be really good.

I don’t have commit access, but I have been looking for meaningful ways to contribute back to the community. Happy to help with code reviews and testing (especially on aarch64 darwin) if needed. I’d also be happy to work on PRs if I manage to get up to speed.

1 Like

As the author of the big bump PR that seems to have kicked all of this off, I’m glad to see people excited to discuss it! As mentioned, please take a look at the PRs linked in the umbrella issue! The big bump, which should address a lot of the outdated packages, has been ready to merge for quite a while now, and I hope it gets merged sometime in the next 3 months.

Once I get a bit more time, I’ll try and put down my thoughts on the various sub-issues linked in the umbrella issue.

1 Like

I think we should add a --version test in an installCheckPhase or something like that

testers.testVersion?

There is no updateScript for JDK derivations.

I do not believe this is “possible”, since not every Java program builds on all JDK.

Also, we have a much more serious problem in my modest opinion: Java bootstrap depends on open source abandonware.
I do not know how much Nixpkgs care about bootstrappability, but I believe this is far from a serious priority.

I am a maintainer of two such packages: jquake and protege-distribution.
While the first one is a proprietary application, I packaged the second like that because, at the time, I couldn’t figure out a way of building complex Maven applications with an acceptable level of maintenance burden on the Nixpkgs side of things.

If we manage to get a good solution for Maven, then maybe I could deprecate that package in favor of a source-built one.

As the maintainer of maptool, I would also be very happy to move to a source-built package if there was a better story for managing Gradle dependencies.

The difference, is that installCheckPhase makes the build fail if --version | grep ${version} fails, whereas testers.testVersion is not being run by nixpkgs-review.

You can try out our (somewhat new) maven.buildMavenPackage.

I’d also encourage you to add a some kind of an installCheckPhase for the proprietary package, so we’ll be able to know via nixpkgs-review if it gets broken without running it ourselves.

1 Like

whereas testers.testVersion is not being run by nixpkgs-review

Is this a bug or a feature of nixpkgs-review?

1 Like

looks like a missing feature Add flag to build package `passthru.tests` by B4dM4n · Pull Request #397 · Mic92/nixpkgs-review · GitHub

1 Like

Even with that PR, or something alike merged, people won’t get used to add the --tests flag to their nixpkgs-review runs, so you might still be missing breakages. I don’t understand this insistence on passthru.tests.testVersion where an installCheckPhase is a much stronger and simpler compliance that a package can fulfill.

1 Like

The Nixpkgs manual says:

Tests that are part of the source package are often executed in the installCheckPhase.

Prefer passthru.tests for tests that are introduced in nixpkgs because:

  • passthru.tests tests the ‘real’ package, independently from the environment in which it was built
  • we can run passthru.tests independently
  • installCheckPhase adds overhead to each build

Should the manual be changed? Or nixpkgs-review? Or people’s habits for how and when they use nixpkgs-review versus other commands?

1 Like

Thanks for quoting Nixpkgs’ manual :slight_smile: . This was discussed a bit at #119731 but people gave their attention in the discussions there on topics other then this. Someone there did raise my opinion, but nobody commented on this opinion…

I noted myself to discuss this at Nixpkgs at some point when I’ll have time.

2 Likes