SBT / Scala use sbt-derivation to build scala file project using flakes

Help
1

I’m trying to make this flakes working for a reproducible build :

  • step 1 / I download and run the sbt script that install scala compiler
  • step 2 / I build the file fetched from a github repo
  • step 3 / I run R script to generate result

I block at Step 1 , sbt return an error when try to download the corresponding scala compiler, i suppose nix/nix-store block the query ?

{
description = "Generate R result from simulation";

inputs = {
    nixpkgs.url = "nixpkgs/nixos-20.09";
    utils.url = "github:numtide/flake-utils";

};

outputs = {self, nixpkgs, utils, mach-nix } : (utils.lib.eachDefaultSystem
    (system :
    let
        pkgs = nixpkgs.legacyPackages.${system};
        REnv = pkgs.rWrapper.override{ packages = with pkgs.rPackages; [tidyverse rmarkdown]; };
        callPackage= nixpkgs.lib.callPackageWith (pkgs // self.packages.${system});

    build5Ad = { stdenv, ... }: stdenv.mkDerivation {
        name = "build5Ad";

        srcs = [
            (builtins.fetchurl {
        url = "https://github.com/sbt/sbt/releases/download/v1.3.10/sbt-1.3.10.tgz";
        sha256 = "1023qcbdbwl40l7mgyd3j0ggr6paz1zs2q787ym52dhrcibhcq1h";
        })
        (builtins.fetchGit {
        url = "https://github.com/eighties-cities/5aday.git";
        rev = "d237c8f327e70938bb23dcfb7fe3fdae73ab2ba5";
         })];

        buildInputs = [
        pkgs.jdk11
        pkgs.scala
         (pkgs.scala.override { jre = pkgs.jdk11; })
        ];

          unpackPhase = ''

            srcs=($srcs)

            unpackFile ''${srcs[0]}

            mkdir -p $out/sbt
            mv sbt/* $out/sbt

            echo "--- running sbt install ---"
            bash $out/sbt/bin/sbt 

          '';

         buildPhase = ''
            export PATH="$out/bin:$PATH"

          '';
    };

    packageName = throw "5Aday";
    in
    {

    defaultPackage = self.packages.${system}.build;

    packages = {
        build = callPackage build5Ad { };
    };

    }


    )
    );

}

@nix { "action": "setPhase", "phase": "unpackPhase" }
unpacking sources
unpacking source archive /nix/store/zpwz745mhlxx9lr9dwfdligbnwxk97i7-sbt-1.3.10.tgz
---
[process_args] java_version = '11'
[sbt_options] declare -a sbt_options=()
[copyRt] java9_rt = '/build/.sbt/0.13/java9-rt-ext-oracle_corporation_11_0_9/rt.jar'
copying runtime jar...
# Executing command line:
java
-Dfile.encoding=UTF-8
-Xms1024m
-Xmx1024m
-Xss4M
-XX:ReservedCodeCacheSize=128m
-Dscala.ext.dirs=/build/.sbt/0.13/java9-rt-ext-oracle_corporation_11_0_9
-jar
/nix/store/q9zy4pam5hsk294plghd5jwci1n3jdfk-build5Ad/sbt/bin/sbt-launch.jar

[info] [launcher] getting org.scala-sbt sbt 1.3.10  (this may take some time)...
You probably access the destination server through a proxy server that is not well configured.
You probably access the destination server through a proxy server that is not well configured.
You probably access the destination server through a proxy server that is not well configured.
You probably access the destination server through a proxy server that is not well configured.
You probably access the destination server through a proxy server that is not well configured.
You probably access the destination server through a proxy server that is not well configured.
You probably access the destination server through a proxy server that is not well configured.
You probably access the destination server through a proxy server that is not well configured.
:

Is there a proxy or something to configure ?
I found few information about scala / sbt and Nix.

2

I found another way using sbt-derivation (GitHub - zaninime/sbt-derivation: mkDerivation for sbt, similar to buildGoModule) with this script, but it seems coursier resolver don’t work and sbt failed to resolve correctly the package…

Any help appreciated… Perhaps @kubukoz or @zaninime ?

{
description = "Generate R result from simulation";

inputs = {
    nixpkgs.url = "nixpkgs/nixos-20.09";
    utils.url = "github:numtide/flake-utils";

    sbt-derivation = {
        type = "github";
        owner = "zaninime";
        repo = "sbt-derivation";
    };

};
outputs = {self, nixpkgs, utils, mach-nix, sbt-derivation } : (utils.lib.eachDefaultSystem
    (system :
    let
       sbtHeadlessOverlay = _: prev: {
         sbt = prev.sbt.override { jre = prev.openjdk11_headless; };
       };

        pkgs = import nixpkgs { inherit system; overlays = [ sbtHeadlessOverlay sbt-derivation.overlay ]; };
        REnv = pkgs.rWrapper.override{ packages = with pkgs.rPackages; [tidyverse rmarkdown]; };
        callPackage= nixpkgs.lib.callPackageWith (pkgs // self.packages.${system});

     buildH24 = { stdenv, ... }: pkgs.sbt.mkDerivation {
            pname = "buildh24";
            version = "1.0.0";

            src = (builtins.fetchGit {
            url = "https://github.com/eighties-cities/h24.git";
            rev = "dad533a6e26d57835af3d24f48179d8709c1e2b1";
            #sha256 = "1rbab9lmqvpg8vdmlkgfbhx1rvq2zxq0yzcrgxsiqqim5qjf6m0k";
          });

            depsSha256 = "I+VfYFQ4SGGpIHiki9i626w71QWcXH8+ZnbB44jShlA=";

          buildInputs = [
            pkgs.coursier
            pkgs.sbt
          ];

             buildPhase = ''
             sbt publishLocal
              '';
        };

    build5Ad = { stdenv, ... }: pkgs.sbt.mkDerivation {
        pname = "build5Ad";
        version = "1.0.0";

        src = (builtins.fetchGit {
        url = "https://github.com/eighties-cities/5aday.git";
        rev = "d237c8f327e70938bb23dcfb7fe3fdae73ab2ba5";
        #sha256 = "1rbab9lmqvpg8vdmlkgfbhx1rvq2zxq0yzcrgxsiqqim5qjf6m0k";
      });

        depsSha256 = "I+VfYFQ4SGGpIHiki9i626w71QWcXH8+ZnbB44jShlA=";

        buildInputs = [
        pkgs.jdk11
        pkgs.scala
         (pkgs.scala.override { jre = pkgs.jdk11; })
        ];

         buildPhase = ''
           sbt
          '';
    };



    buildRScripts = { stdenv, fetch,... }: stdenv.mkDerivation {

        name = "myscript";
        src = self;
        nativeBuildInputs = [ REnv ];
        dontBuild = true;
        buildInputs = [ pkgs.pandoc pkgs.unzip ];

        #bash ./download.sh UAPhIVpjONxBvkf84jrBqtmm2elgZrAcAVH4NopS0DS4loM0bJGgE4LXIhhL
        #${REnv}/bin/Rscript -e 'rmarkdown::render("src/R/test.Rmd")'
        installPhase=''
            mkdir $out
            cd $out
            unzip ${fetch}/*.zip
        '';
    };
    packageName = throw "5Aday";
    in
    {

    defaultPackage = self.packages.${system}.buildH24;

    packages = {
        buildH24 = callPackage buildH24 { };
        build5Ad = callPackage build5Ad { };
    };

    }


    )
    );

}
3

Hi, I looked into your derivation and I found few things that don’t really make sense to me.

  • build5Ad has some buildInputs, all of them shouldn’t be necessary, scala is even mentioned twice? If you want a specific jdk version you should have a global overlay instead, as that’s what sbt.mkDerivation will use.
  • buildH24 is a function from stdenv, but it’s not used anywhere, it can be simplified, simply remove the { stdenv, ... }: part. Also the buildInputs there are not needed usually.

For the rest, I don’t know exactly what’s failing as it’s not mentioned. sbt uses coursier since 1.3.0 and I have many projects that are using it with sbt-derivation without any problems.

1 Like
4

Thanks !

You’re right, my example is not clear because i make some typo during test and i try multiple things like overlay / buildInputs due to my misunderstanding on some nixos things …

build5Ad need buildH24 as library published locally on .ivy to correctly compile, i don’t know if this is possible. Sbt compile h24 and publish locally, then 5Ad is compiled using this library.

build5Ad = { stdenv, buildH24 }: pkgs.sbt.mkDerivation {

I have error of resolution for some artifacts when i run the flake, error which don’t happen running sbt localy, outside of this flake. I take the time to rewrote a new version using your remarks tomorrow !

Update for @zaninime : Running just sbt publishLocal of H24 project, sbt failed with this error, the maven classic repo is malformed and don’t contain the jar, so normally resolver try another valid repo (existing in the build.sbt project, see h24/build.sbt at master · eighties-cities/h24 · GitHub) when i run sbt locally, but with sbt-derivation this is not the case. Is there an option ?

error: builder for '/nix/store/ac54jx2gbvggdcacmp8bwfpg40plbmha-buildh24-1.0.0.drv' failed with exit code 1;
       last 10 log lines:
       > [error]   Not found
       > [error]   Not found
       > [error]   not found: /build/source/.nix/ivy/localjavax.media/jai-core/1.1.3/ivys/ivy.xml
       > [error]   download error: Caught java.net.UnknownHostException (repo1.maven.org) while downloading https://repo1.maven.org/maven2/javax/media/jai-core/1.1.3/jai-core-1.1.3.pom
       > [error]   download error: Caught java.net.UnknownHostException (repo.osgeo.org) while downloading https://repo.osgeo.org/repository/release/javax/media/jai-core/1.1.3/jai-core-1.1.3.pom
       > [error]   download error: Caught java.net.UnknownHostException (maven.geo-solutions.it) while downloading https://maven.geo-solutions.it/javax/media/jai-core/1.1.3/jai-core-1.1.3.pom
       > [error]   download error: Caught java.net.UnknownHostException (maven.geotoolkit.org) while downloading https://maven.geotoolkit.org/javax/media/jai-core/1.1.3/jai-core-1.1.3.pom
       > [error]   download error: Caught java.net.UnknownHostException (repo.boundlessgeo.com) while downloading https://repo.boundlessgeo.com/main/javax/media/jai-core/1.1.3/jai-core-1.1.3.pom
       > [error]   jai_core-1.1.3.jar not found under https://repo.osgeo.org/repository/release/javax/media/jai_core/1.1.3/
       > [error] Total time: 1 s, completed Jan 26, 2022, 9:32:23 AM
5

This is due to the build sandbox. To verify try adding

    outputHashAlgo = "sha256";                                                                                                                                                        
    outputHash = "3d461fd46b3d645e33fb1d23c5c27741f2c3b51f055cf69f0a8b364af33b9c32";

To the derivation then building. The build should still fail but at least progress beyond fetching dependencies.

This results in nix treating this as a fixed-output derivation . Which is permitted to access the network: