Gradle2nix V2: Call for testers

[tad]

gradle2nix V2 is a complete rewrite using a new, faster, and hopefully more reliable method of extracting artifact information from Gradle. It has been tested on the jetbrains/kotlin repo and has successfully compiled Kotlin from source, which is not a small feat given the size and complexity of that project.

Please give it a try; it’s in the v2 branch, and a PR with a summary of changes is in #62. I’m going to integrate it with my main project at my day job, which should provide sufficient testing for a release, but if more folk tested this then I would feel a bit better about shipping it.

[roberth]

I’m not a gradle user, but please steal this trick from sbtix.
It lets you use the Nix store as a content addressable store so that you’re only copying the actual new jars when copying closures.
(if it’s not already efficient like that; not sure)

[collinarnett]

Oh this is so amazing! I’m currently working with Datahub which is a nightmare to work with at the moment because of gradle.

[tad]

The local repo is built with symlinkJoin, which basically does the same thing.

[roberth]

That sounds good, but it seems that that happens before the build, iiuc. Is deduplication also applied to the output of the derivation that performs the actual build?

[tad]

It looks like sbtix needs that because they support locally-built dependencies by copying them to the local repo. gradle2nix doesn’t do any copying of artifacts, just symlinking, but it’s a good note for the future if we want to handle incremental builds.

[RCoeurjoly]

Nice work! For poetry2nix, I created a project that attempts to install each project from a list of the 4000 most downloaded Python packages. Interestingly, I found that about a third of them could not be installed.

I suggest doing something similar for gradle2nix. Although I couldn’t find a comprehensive list of the most downloaded Gradle projects with a quick search, I’m sure you would have better luck as a Gradle user. It could be an insightful exercise to identify and address compatibility issues early on.

[numinit]

Hey @tad! Thank you for your work. Projects like this make it much more realistic for me to do Nix-based Gradle builds of our products for my job.

For some background, I work on an application team at a medium to large size company, and have always been hampered by the sandbox on our Android builds because using Gradle in the sandbox has been annoying. I’m currently working on fully sandboxing a few other very complicated builds, one of which merits a blogpost in the future for the utter craziness you can get up to with Nix. In the meantime, though, here’s a sanitized review of my progress with gradle2nix so far.

---

The first thing I ran into yesterday was determining whether or not to use Gradle’s dependency locking. I realized after most of a day that this was basically futile. The Gradle lockfile doesn’t include URLs or hashes, but does include versions, making it basically useless to Nix.

So, I started today by just using gradle2nix. Since I had an older pre-flake shell.nix that had not been converted to a flake, I started there. That part was all fairly straightforward, especially because I had just updated all our project’s dependencies to meet the latest and greatest Gradle 8.8.

The first thing I had to do was come up with all the tasks to run in order to spit out the correct set of dependencies, since I could not figure out how to make Gradle spit them out itself, even though that is part of its job, and the default would not work for our large multi-module project. Considering that Gradle is a non-functional (in every single sense of the word), Turing-complete language where these can be literally anything depending on the current Zodiac sign, cosmological alignment, or phase of the moon, I lamented for humanity’s future for reproducible builds for a very brief moment and then got to work.

For our (rather large) corporate-scale Android Gradle project, this looks something like:

• Build
• Lint
• Test
• Android instrumented test

So I got to work adding a gradle-lock script to our Nix devShell that would run all these via the gradle2nix wrapper using the right JAVA_HOME.

The build was easy enough; I just had to add assembleDebug and assembleRelease to the list of jobs. So was lint, and our unit tests. Android instrumented tests wanted a physical device or emulator to be working, so I settled for something similar to packageDebugAndroidTest. Presumably Gradle will have figured out everything it depends on by then.

This was easy enough and I got a nearly 4000 line JSON file containing all of my project’s possible dependencies, in every month, Zodiac sign, or phase of the moon. I instantiated this using buildGradleApplication (knowing damn well this wouldn’t be enough), and…

Oof. Two private repositories that require auth token HTTP headers set, so it can’t download those jars. Two days in, and that’s where I am, so I’ll be debugging this next.

---

This seems pretty promising and was easier to set up than I thought, especially considering the detour with Gradle’s lockfiles. I’ve got a couple considerations:

• Gradle’s native lockfile mechanism is next to useless for this and this isn’t documented anywhere.
• Fetching things that require API keys is hard, but it always has been.
• Gradle compatibility. Presumably Gradle will change something else and this approach will break again in the future.
• Probably something involving the gradle wrapper that will force me to parse a gradle-wrapper.properties file for an Android project in bash, but I’m committing to the bit at this point and will give it a shot.

Overall, thanks for a great project. My mental model of how this works at this point is that it hooks Gradle somewhere so it can collect up all the dependencies and write them to a lockfile at the end.

[tad]

Thanks! I’m also an Android engineer, so I understand your pain.

@numinit

Oof. Two private repositories that require auth token HTTP headers set, so it can’t download those jars. Two days in, and that’s where I am, so I’ll be debugging this next.

I need to document this, but buildGradlePackage takes a fetchers attrset of scheme → fetch function, where a fetch function is like:

{ url, hash }: <derivation>

In your case, there is a hack/workaround in the wiki, so I think the right incantations needed are:

1. Override the https fetcher like so:

buildGradlePackage {
  # ...
  fetchers = {
    https = if lib.hasPrefix "https://my-secure-host.com" then stdenv.fetchurlBoot else pkgs.fetchurl;
  }
}

2. Add the following to /etc/nix/netrc:

machine DOMAINNAME
    login USERNAME
    password SECRET

3. Add this to your flake:

{
  # inputs, outputs, ...
  nixConfig = {
    netrc-file = "/etc/nix/netrc";
  };
}

Hopefully this leads to something that works for your project.

@numinit

Gradle compatibility. Presumably Gradle will change something else and this approach will break again in the future.

Likely, but there will always be workarounds. The most recent change I’ve made is to produce version-specific Gradle plugins so we can adapt to API changes.

@numinit

Probably something involving the gradle wrapper that will force me to parse a gradle-wrapper.properties file for an Android project in bash, but I’m committing to the bit at this point and will give it a shot.

I highly suggest either not using the wrapper, or always using a version that’s available in nixpkgs. Failing that, pkgs.gradleGen is what you need to make your own nix-flavored Gradle:

{ lib, callPackage, gradleGen, jdk }:
callPackage (gradleGen {
  version = "8.7";
  hash = "sha256-VEw11r2Emuil7QvOo5umd9xA9J330YNVYVgtogCblh0=";
  nativeVersion = "0.22-milestone-25";
  defaultJava = jdk;
})

Here’s an update script that will prefetch those values for you and dump them in a source.json in your repo:

#!/usr/bin/env bash
set -e -o pipefail

cd "$(git rev-parse --show-toplevel)"
pkg="$(realpath "$(dirname "$0")")"

version="$1"
prefetch="$(nix-prefetch-url "https://services.gradle.org/distributions/gradle-${version}-bin.zip" --print-path)"
hash="$(echo "$prefetch" | head -n 1 | xargs nix hash to-sri --type sha256 | head -n 1)"
path="$(echo "$prefetch" | tail -n 1)"
nativeVersion="$(zipinfo -1 "$path" | xargs basename -s .jar | grep 'native-platform-[0-9].*' | grep -o '[0-9].*')"

nix run nixpkgs#jq -- --arg version "$version" --arg hash "$hash" --arg nativeVersion "$nativeVersion" -n '$ARGS.named' > "$pkg/source.json"
cd -

@numinit

Overall, thanks for a great project. My mental model of how this works at this point is that it hooks Gradle somewhere so it can collect up all the dependencies and write them to a lockfile at the end.

Thank you! That’s essentially what it does; we use the Tooling API to run the build with a Gradle plugin. The plugin listens to Gradle’s build operations log and gathers up all events that are pinging dependency URLs. Finally, we loop through the URLs and look up its Gradle cache entry, which points to the file on disk and lets us calculate a SHA-256. Overall, very simple and much faster than trying to resolve dependency artifacts outside of Gradle.

[numinit]

@tad

In your case, there is a hack/workaround in the wiki

This has worked just fine so far.

[GetPsyched]

Thanks a ton for this effort @tad.

I maintain the ATLauncher package in nixpkgs and have been meaning to build it from source ever since I introduced it. This makes me really excited to test out gradle2nix v2.

I tried it on the project, built using a very simple script for now and I’ve seem to have run into some errors! Good news, bad news, it’s all perspective. Anyways, I have raised an issue about this on the repo. See you there

[numinit]

@tad Nearly 2 weeks in, here’s a re-review on a complex enterprise Android project with a multi-flavor build step, tests, on-device instrumented tests, and lint, with all the new setup hook changes.

TL;DR: This is the first tool in 5 years of using Nix that has made it fully realistic to build one of our products in the Nix sandbox, and even works with ADB on-device Android tests if I use __noChroot = true in the derivation, plus sandbox = relaxed and extra-sandbox-paths = /dev with a udev rule to apply ACLs to the USB devices so the nixbld group can read and write to them. Access to a release signing keystore works in a similar way, with “well-known” paths passed into the sandbox.

Breaking it down a bit: I think the new setup hook changes are overwhelmingly good. I structured my usage of gradle2nix as a function that returns a different derivation based on whether I’d like to run the build, lint, tests, or Android tests. I don’t bother with installPhase because I have a custom installPhase that finds all the build artifacts for the requested output (APKs and obfuscation maps from buildPhase, or test/lint result XMLs from checkPhase) and copies them all to $out.

(I realize that I could combine all these into a build/check in a single derivation, but couldn’t find a good answer. When do tests run vs lint? They’re both “checks,” basically, and I may revisit this.)

The one bit of jank I ran into was gradleCheckTasks and gradleCheckFlags. I think this condition is misleading and should take into account whether gradleCheckTasks is set. I saw a similar pattern in the installPhase too, and also noticed an asymmetry with gradleBuildPhase (i.e. there’s no gradleBuildTasks so I have to pass the build tasks I’d like in as gradleBuildFlags.

My current solution to make sure that my checks run is ensuring that a dummy ["--console=plain"] is passed in so the gradleCheckFlags are always non-empty (which is another thing I noticed, that may want to be a default setting because otherwise the build output gets garbled).

Thanks once again for the work on this!

[tad]

I’m happy it’s working out for you!

@numinit

The one bit of jank I ran into was gradleCheckTasks and gradleCheckFlags. I think this condition is misleading and should take into account whether gradleCheckTasks is set. I saw a similar pattern in the installPhase too, and also noticed an asymmetry with gradleBuildPhase (i.e. there’s no gradleBuildTasks so I have to pass the build tasks I’d like in as gradleBuildFlags.

Yeah, I think removing the *Tasks variables altogether is the right move here. I used to have code that ran gradle help --task=$task to check if a task exists, but it was unbearably slow, and the *Tasks variables are a remnant of that. WDYT?

[numinit]

@tad

Yeah, I think removing the *Tasks variables altogether is the right move here. I used to have code that ran gradle help --task=$task to check if a task exists, but it was unbearably slow, and the *Tasks variables are a remnant of that. WDYT?

My usecase for them was running a Gradle task other than check (which is the default) as the checkPhase. As long as that’s customizable somehow (and I can switch which check task I’m running), that sounds fine.

Maybe default the flags to ["check"], though we’d have to figure out another condition under which to run the checkPhase. Maybe doCheck = true; and doInstall = true; are sufficient for each of the respective phases, and this would be a little more intuitive?

[edelagnier]

Looking forward to the release.
It may be an obvious mistake but when trying to install it, I had the following error :

[me@xiangu:~/code/app]$ nix build -f "https://github.com/tadfisher/gradle2nix/archive/master.tar.gz" --extra-experimental-features nix-command
error:
       … while calling the 'derivationStrict' builtin

         at /builtin/derivation.nix:9:12: (source not available)

       … while evaluating derivation 'gradle2nix-1.0.0-rc2'
         whose name attribute is located at /nix/store/mxsf7ix3w0bhnr79ds9gfq4pga59j8jd-nixos-24.05/nixos/pkgs/stdenv/generic/make-derivation.nix:331:7

       … while evaluating attribute 'buildPhase' of derivation 'gradle2nix-1.0.0-rc2'

         at /nix/store/ik3jszcx5nwbbnaygkf2kmsy232alf98-source/gradle-env.nix:380:3:

          379|
          380|   buildPhase = args.buildPhase or ''
             |   ^
          381|     runHook preBuild

       (stack trace truncated; use '--show-trace' to show the full trace)

       error: value is a function while a set was expected

Would you know why ?

[tad]

Yes, you’ll need to use the v2 branch, e.g.

nix build -f "https://github.com/tadfisher/gradle2nix/archive/v2.tar.gz" --extra-experimental-features nix-command

[edelagnier]

ok… obvious indeed. Thanks !

I thougt about updating the Readme.md but I expect it will soon be osbolete when the branch is merged in master.

I had trouble in the next step.
Running gradle2nix -t uberjar command generated the expected gradle.lock

I then tried to create package :

backend.nix :

{ pkgs ? import <nixpkgs> {} }:

let
  gradle2nix = import (fetchTarball "https://github.com/tadfisher/gradle2nix/archive/v2.tar.gz")  {};
in
gradle2nix.buildGradlePackage {
  pname = "backend";
  version = "1.0";
  lockFile = ./gradle.lock;
  gradleInstallFlags = [ "uberjar" ];
}

running

nix-build backend.nix

the error message :

Running phase: unpackPhase
variable $src or $srcs should point to the source

I tried to add a src variable pointing to the resulting jar but got another error

Running phase: unpackPhase
unpacking source archive /nix/store/r85ryvyc1ykk8whmpp8bv7nr89n4q23q-backend-uber.jar
do not know how to unpack source archive /nix/store/r85ryvyc1ykk8whmpp8bv7nr89n4q23q-backend-uber.jar

Sorry if it’s also an obvious answer, at least the doc will be beginner proof

[tad]

src should point to the source code of what you’re building, using something like fetchgit, fetchFromGithub, or a local file path in the repo. Don’t point it at a JAR, that’s already built locally.

[Tmplt]

V2 build fails currently:

$ nix build -L -f "https://github.com/tadfisher/gradle2nix/archive/v2.tar.gz"
gradle2nix> FAILURE: Build failed with an exception.
gradle2nix> * Where:
gradle2nix> Build file '/build/source/buildSrc/build.gradle.kts' line: 1
gradle2nix> * What went wrong:
gradle2nix> Plugin [id: 'org.gradle.kotlin.kotlin-dsl', version: '4.3.0'] was not found in any of the following sources:
gradle2nix> - Gradle Core Plugins (plugin is not in 'org.gradle' namespace)
gradle2nix> - Included Builds (No included builds contain this plugin)
gradle2nix> - Plugin Repositories (could not resolve plugin artifact 'org.gradle.kotlin.kotlin-dsl:org.gradle.kotlin.kotlin-dsl.gradle.plugin:4.3.0')
gradle2nix>   Searched in the following repositories:
gradle2nix>     Gradle Central Plugin Repository(file:/nix/store/x61mq8b567ckfsh9bpsyssds8h7lcg2j-gradle-maven-repo/)

[numinit]

Is there an easier way to update dependencies than fully running through all builds for all relevant tasks? Months in, and that’s my only gripe.