Gradle2nix V2: Call for testers

gradle2nix V2 is a complete rewrite using a new, faster, and hopefully more reliable method of extracting artifact information from Gradle. It has been tested on the jetbrains/kotlin repo and has successfully compiled Kotlin from source, which is not a small feat given the size and complexity of that project.

Please give it a try; it’s in the v2 branch, and a PR with a summary of changes is in #62. I’m going to integrate it with my main project at my day job, which should provide sufficient testing for a release, but if more folk tested this then I would feel a bit better about shipping it.

15 Likes

I’m not a gradle user, but please steal this trick from sbtix.
It lets you use the Nix store as a content addressable store so that you’re only copying the actual new jars when copying closures. :rocket:
(if it’s not already efficient like that; not sure)

Oh this is so amazing! I’m currently working with Datahub which is a nightmare to work with at the moment because of gradle.

The local repo is built with symlinkJoin, which basically does the same thing.

That sounds good, but it seems that that happens before the build, iiuc. Is deduplication also applied to the output of the derivation that performs the actual build?

It looks like sbtix needs that because they support locally-built dependencies by copying them to the local repo. gradle2nix doesn’t do any copying of artifacts, just symlinking, but it’s a good note for the future if we want to handle incremental builds.

Nice work! For poetry2nix, I created a project that attempts to install each project from a list of the 4000 most downloaded Python packages. Interestingly, I found that about a third of them could not be installed.

I suggest doing something similar for gradle2nix. Although I couldn’t find a comprehensive list of the most downloaded Gradle projects with a quick search, I’m sure you would have better luck as a Gradle user. It could be an insightful exercise to identify and address compatibility issues early on.

Hey @tad! Thank you for your work. Projects like this make it much more realistic for me to do Nix-based Gradle builds of our products for my job.

For some background, I work on an application team at a medium to large size company, and have always been hampered by the sandbox on our Android builds because using Gradle in the sandbox has been annoying. I’m currently working on fully sandboxing a few other very complicated builds, one of which merits a blogpost in the future for the utter craziness you can get up to with Nix. In the meantime, though, here’s a sanitized review of my progress with gradle2nix so far.


The first thing I ran into yesterday was determining whether or not to use Gradle’s dependency locking. I realized after most of a day that this was basically futile. The Gradle lockfile doesn’t include URLs or hashes, but does include versions, making it basically useless to Nix.

So, I started today by just using gradle2nix. Since I had an older pre-flake shell.nix that had not been converted to a flake, I started there. That part was all fairly straightforward, especially because I had just updated all our project’s dependencies to meet the latest and greatest Gradle 8.8.

The first thing I had to do was come up with all the tasks to run in order to spit out the correct set of dependencies, since I could not figure out how to make Gradle spit them out itself, even though that is part of its job, and the default would not work for our large multi-module project. Considering that Gradle is a non-functional (in every single sense of the word), Turing-complete language where these can be literally anything depending on the current Zodiac sign, cosmological alignment, or phase of the moon, I lamented for humanity’s future for reproducible builds for a very brief moment and then got to work.

For our (rather large) corporate-scale Android Gradle project, this looks something like:

  • Build
  • Lint
  • Test
  • Android instrumented test

So I got to work adding a gradle-lock script to our Nix devShell that would run all these via the gradle2nix wrapper using the right JAVA_HOME.

The build was easy enough; I just had to add assembleDebug and assembleRelease to the list of jobs. So was lint, and our unit tests. Android instrumented tests wanted a physical device or emulator to be working, so I settled for something similar to packageDebugAndroidTest. Presumably Gradle will have figured out everything it depends on by then.

This was easy enough and I got a nearly 4000 line JSON file containing all of my project’s possible dependencies, in every month, Zodiac sign, or phase of the moon. I instantiated this using buildGradleApplication (knowing damn well this wouldn’t be enough), and…

Oof. Two private repositories that require auth token HTTP headers set, so it can’t download those jars. Two days in, and that’s where I am, so I’ll be debugging this next.


This seems pretty promising and was easier to set up than I thought, especially considering the detour with Gradle’s lockfiles. I’ve got a couple considerations:

  • Gradle’s native lockfile mechanism is next to useless for this and this isn’t documented anywhere.
  • Fetching things that require API keys is hard, but it always has been.
  • Gradle compatibility. Presumably Gradle will change something else and this approach will break again in the future. :frowning:
  • Probably something involving the gradle wrapper that will force me to parse a gradle-wrapper.properties file for an Android project in bash, but I’m committing to the bit at this point and will give it a shot.

Overall, thanks for a great project. My mental model of how this works at this point is that it hooks Gradle somewhere so it can collect up all the dependencies and write them to a lockfile at the end.

Thanks! I’m also an Android engineer, so I understand your pain.

I need to document this, but buildGradlePackage takes a fetchers attrset of scheme → fetch function, where a fetch function is like:

{ url, hash }: <derivation>

In your case, there is a hack/workaround in the wiki, so I think the right incantations needed are:

  1. Override the https fetcher like so:
buildGradlePackage {
  # ...
  fetchers = {
    https = if lib.hasPrefix "https://my-secure-host.com" then stdenv.fetchurlBoot else pkgs.fetchurl;
  }
}
  1. Add the following to /etc/nix/netrc:
machine DOMAINNAME
    login USERNAME
    password SECRET
  1. Add this to your flake:
{
  # inputs, outputs, ...
  nixConfig = {
    netrc-file = "/etc/nix/netrc";
  };
}

Hopefully this leads to something that works for your project.

Likely, but there will always be workarounds. The most recent change I’ve made is to produce version-specific Gradle plugins so we can adapt to API changes.

I highly suggest either not using the wrapper, or always using a version that’s available in nixpkgs. Failing that, pkgs.gradleGen is what you need to make your own nix-flavored Gradle:

{ lib, callPackage, gradleGen, jdk }:
callPackage (gradleGen {
  version = "8.7";
  hash = "sha256-VEw11r2Emuil7QvOo5umd9xA9J330YNVYVgtogCblh0=";
  nativeVersion = "0.22-milestone-25";
  defaultJava = jdk;
})

Here’s an update script that will prefetch those values for you and dump them in a source.json in your repo:

#!/usr/bin/env bash
set -e -o pipefail

cd "$(git rev-parse --show-toplevel)"
pkg="$(realpath "$(dirname "$0")")"

version="$1"
prefetch="$(nix-prefetch-url "https://services.gradle.org/distributions/gradle-${version}-bin.zip" --print-path)"
hash="$(echo "$prefetch" | head -n 1 | xargs nix hash to-sri --type sha256 | head -n 1)"
path="$(echo "$prefetch" | tail -n 1)"
nativeVersion="$(zipinfo -1 "$path" | xargs basename -s .jar | grep 'native-platform-[0-9].*' | grep -o '[0-9].*')"

nix run nixpkgs#jq -- --arg version "$version" --arg hash "$hash" --arg nativeVersion "$nativeVersion" -n '$ARGS.named' > "$pkg/source.json"
cd -

Thank you! That’s essentially what it does; we use the Tooling API to run the build with a Gradle plugin. The plugin listens to Gradle’s build operations log and gathers up all events that are pinging dependency URLs. Finally, we loop through the URLs and look up its Gradle cache entry, which points to the file on disk and lets us calculate a SHA-256. Overall, very simple and much faster than trying to resolve dependency artifacts outside of Gradle.

1 Like

This has worked just fine so far. :slight_smile:

Thanks a ton for this effort @tad.

I maintain the ATLauncher package in nixpkgs and have been meaning to build it from source ever since I introduced it. This makes me really excited to test out gradle2nix v2.

I tried it on the project, built using a very simple script for now and I’ve seem to have run into some errors! Good news, bad news, it’s all perspective. Anyways, I have raised an issue about this on the repo. See you there :stuck_out_tongue: