How do you discover the override attributes for a derivation?

Help
1

I’m trying to override the jetbrains idea-ultimate-2019-2.4 derivation as it depends upon a non-existing derivation jbrsdk-11_0_2-osx-x64-b485.1.tar.gz.drv.

I had assumed that overriding an attribute would do just that and not download the broken package.

How do I discover where it is still being depended and override it?
Can I list the computed overridable attributes somehow?

What I have so far is this:

{
  allowUnfree = true;
  packageOverrides = pkgs: with pkgs; rec {

    myJetbrainsJdk = pkgs.jetbrains.jdk.overrideAttrs (oldAttrs: rec {
      version = "520.11";
      src = pkgs.fetchurl {
        url = "https://bintray.com/jetbrains/intellij-jdk/download_file?file_path=jbrsdk-11_0_4-osx-x64-b520.11.tar.gz";
        sha256 = "0d1qwbssc8ih62rlfxxxcn8i65cjgycdfy1dc1b902j46dqjkq9z";
      };
    });
    
    myIdeaUltimate = pkgs.jetbrains.idea-ultimate.override {
      jdk = myJetbrainsJdk;
    };

However, even thought I’ve overridden the jdk attribute, it still seems to want to include the broken derivation in the list of derivations to be build.

nix-channel --update; nix-env -iA nixpkgs.myPackages
unpacking channels...
replacing old 'my-packages'
installing 'my-packages'
these derivations will be built:
  /nix/store/9kfi3k9q6hi7z3lwann318hndbah535v-idea-ultimate.desktop.drv
  /nix/store/ica1m5yq3f3y05xnw7ln1lnfvp0yjvyf-download_file?file_path=jbrsdk-11_0_4-osx-x64-b520.11.tar.gz.drv
  /nix/store/bf2hwhrvfl8g77gdiw053rayh06x0120-jetbrainsjdk-520.11.drv
  /nix/store/fazsa1a4l70s391rjk9yyi2hvrg0zbmp-download_file?file_path=jbrsdk-11_0_2-osx-x64-b485.1.tar.gz.drv
  /nix/store/fwwk976sd278zb68zy9wm5pkxss0rnhg-jetbrainsjdk-485.1.drv
  /nix/store/s3m2bpcyrnx9dcq4drh95882n0mk1d6m-ideaIU-2019.2.4-no-jbr.tar.gz.drv
  /nix/store/9kiajpmmsp3i6ysj4vdqq8dzi84mnr73-idea-ultimate-2019.2.4.drv
  /nix/store/jh1ixm54qinv8pk6kypvv6n6cfr4sws8-my-packages.drv
these paths will be fetched (0.02 MiB download, 0.12 MiB unpacked):
  /nix/store/hp90sbwznq1msv327f0lb27imvcvi80h-libnotify-0.7.8
building '/nix/store/9kfi3k9q6hi7z3lwann318hndbah535v-idea-ultimate.desktop.drv'...
copying path '/nix/store/hp90sbwznq1msv327f0lb27imvcvi80h-libnotify-0.7.8' from 'https://cache.nixos.org'...
building '/nix/store/fazsa1a4l70s391rjk9yyi2hvrg0zbmp-download_file?file_path=jbrsdk-11_0_2-osx-x64-b485.1.tar.gz.drv'...

trying https://bintray.com/jetbrains/intellij-jdk/download_file?file_path=jbrsdk-11_0_2-osx-x64-b485.1.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (22) The requested URL returned error: 404 Not Found
error: cannot download download_file?file_path=jbrsdk-11_0_2-osx-x64-b485.1.tar.gz from any mirror
builder for '/nix/store/fazsa1a4l70s391rjk9yyi2hvrg0zbmp-download_file?file_path=jbrsdk-11_0_2-osx-x64-b485.1.tar.gz.drv' failed with exit code 1
building '/nix/store/ica1m5yq3f3y05xnw7ln1lnfvp0yjvyf-download_file?file_path=jbrsdk-11_0_4-osx-x64-b520.11.tar.gz.drv'...
cannot build derivation '/nix/store/fwwk976sd278zb68zy9wm5pkxss0rnhg-jetbrainsjdk-485.1.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/9kiajpmmsp3i6ysj4vdqq8dzi84mnr73-idea-ultimate-2019.2.4.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/jh1ixm54qinv8pk6kypvv6n6cfr4sws8-my-packages.drv': 1 dependencies couldn't be built
error: build of '/nix/store/jh1ixm54qinv8pk6kypvv6n6cfr4sws8-my-packages.drv' failed
2 Likes
2

I’ve just tried your snippet and it doesn’t download two version of JDK:

$ nix-env -iA nixpkgs.myIdeaUltimate --dry-run
installing 'idea-ultimate-2019.1.1'
these derivations will be built:
  /nix/store/11ghwdcpfzbqxccxfyw7fjh700d7rzf7-download_file?file_path=jbrsdk-11_0_4-osx-x64-b520.11.tar.gz.drv
  /nix/store/27aqwz57xnrjsba8zsh111lm37v2mm2l-ideaIU-2019.1.1-no-jbr.tar.gz.drv
  /nix/store/69xm1bfgqh4c2pcf2w3mj26ngbcblj99-jetbrainsjdk-164.drv
  /nix/store/fscjb3f7pg566v3xdn759s9pnkr0mszs-idea-ultimate-2019.1.1.drv
these paths will be fetched (169.02 MiB download, 402.18 MiB unpacked):
  /nix/store/659si0x9cf4fywfy0c6ihw05rx57r9qn-jetbrainsjdk-164
  /nix/store/ihppkwjq3zvgsc2fgkajgx0dp3v4v6ll-p7zip-16.02
  /nix/store/j4gw53x4cn8dqycmgkxc3g1vxkibj70g-libav-0.8.21

So I think something else uses old JDK.

You may try to replace it with overlays.

self: super: {
  jetbrains = super.jetbrains // {
    jdk = super.jetbrains.jdk.overrideAttrs (...);
  };
}

It should override JDK for jetbrains.idea-ultimate, because of how the knot is tied:

  jetbrains = (recurseIntoAttrs (callPackages ../applications/editors/jetbrains {
    jdk = jetbrains.jdk;
  }) // {
    jdk = callPackage ../development/compilers/jetbrains-jdk {  };
  });
3

Thanks for the suggestion @danbst I’ve tried your suggestion (if I understood correctly), but must be missing something.

self: super: {
  jetbrains = super.jetbrains {
    jdk = super.jetbrains.jdk.overrideAttrs (oldAttrs: rec {
      version = "520.11";
      src = super.fetchurl {
        url = "https://bintray.com/jetbrains/intellij-jdk/download_file?file_path=jbrsdk-11_0_4-osx-x64-b520.11.tar.gz";
	sha256 = "0d1qwbssc8ih62rlfxxxcn8i65cjgycdfy1dc1b902j46dqjkq9z";
      };
    });
  };
}

but it complains

nix-env -i 'idea-ultimate' --dry-run
(dry run; not doing anything)
error: attempt to call something which is not a function but a set, at /Users/ldeck/.config/nixpkgs/overlays/02-jetbrains.nix:2:15
4

should be jetbrains = super.jetbrains // {. It is because super.jetbrains is an attrset, and I replace a jdk attribute in this attrset.

1 Like
5

Ah, I’d assumed that was a discourse formatting problem rather than intentional :slight_smile:.

Nevertheless, @danbst it’s still wanting to install the invalid jdk jbrsdk-11_0_2-osx-x64-b485.1.tar.gz.drv.

Any ideas why that would be?

nix-env -iA 'nixpkgs.jetbrains.idea-ultimate' --dry-run
(dry run; not doing anything)
installing 'idea-ultimate-2019.2.4'
these derivations will be built:
  /nix/store/ica1m5yq3f3y05xnw7ln1lnfvp0yjvyf-download_file?file_path=jbrsdk-11_0_4-osx-x64-b520.11.tar.gz.drv
  /nix/store/bf2hwhrvfl8g77gdiw053rayh06x0120-jetbrainsjdk-520.11.drv
  /nix/store/fazsa1a4l70s391rjk9yyi2hvrg0zbmp-download_file?file_path=jbrsdk-11_0_2-osx-x64-b485.1.tar.gz.drv
  /nix/store/fwwk976sd278zb68zy9wm5pkxss0rnhg-jetbrainsjdk-485.1.drv
  /nix/store/9kiajpmmsp3i6ysj4vdqq8dzi84mnr73-idea-ultimate-2019.2.4.drv

Is it because overrideAttrs firstly calculates the super’s attributes before overriding them?

6

if it helps, at all @danbst, my system info is as follows:

nix run nixpkgs.nix-info -c nix-info -m
[2 copied (0.0 MiB), 0.0 MiB DL]
 - system: `"x86_64-darwin"`
 - host os: `Darwin 19.2.0, macOS 10.15.2`
 - multi-user?: `no`
 - sandbox: `no`
 - version: `nix-env (Nix) 2.3.1`
 - channels(ldeck): `"nixpkgs-20.03pre205792.fb1bc1b891f"`
 - nixpkgs: `/Users/ldeck/.nix-defexpr/channels/nixpkgs`
7

Let’s make it fully reproducible:

let
  nixpkgs = builtins.fetchTarball "https://github.com/nixos/nixpkgs/archive/4d2dd155461.tar.gz";
  pkgs = import nixpkgs {
      config.allowUnfree = true;
      overlays = [
          (self: super: {
            jetbrains = super.jetbrains // {
                jdk = super.jetbrains.jdk.overrideAttrs (oldAttrs: rec {
                    version = "520.11";
                    src = super.fetchurl {
                        url = "https://bintray.com/jetbrains/intellij-jdk/download_file?file_path=jbrsdk-11_0_4-osx-x64-b520.11.tar.gz";
                        sha256 = "0d1qwbssc8ih62rlfxxxcn8i65cjgycdfy1dc1b902j46dqjkq9z";
                    };
                });
            };
          })
      ];
  };
in pkgs.jetbrains.idea-ultimate

When I build it it doesn’t use old sources at all. But fails on fetch step (cannot download that file).

8

It’s there on bintray though @danbst.
See Service End for Bintray, JCenter, GoCenter, and ChartCenter | JFrog

9

Hmm, looks like I’ve provided the wrong sha256 though. It should be 3fe1297133440a9056602d78d7987f9215139165bd7747b3303022a6f5e23834

Perhaps that’ll help? @danbst

10

btw, I’ve reproduced the error. I was wrong, it indeed downloads old jdk version as well (but on linux it downloads it from binary cache). I’ll look into

2 Likes
11

@danbst how do you run that as a standalone reproducible thing?

12

save it to file.nix and run nix-build that/file.nix

1 Like
13

cool, thanks. Yes, here’s the complete output for me:

14

Okay, I found a bug in Nixpkgs. Made a PR. You can use that in a overlay like:

let
  pkgs = import <nixpkgs> {
      config.allowUnfree = true;
      overlays = [
          (self: super:
          let danbstFixedNixpkgs = builtins.fetchTarball https://github.com/danbst/nixpkgs/archive/203a8d12be0cf5609.tar.gz;
              danbstFixedPkgs = import danbstFixedNixpkgs { };
          in {
            jetbrains = super.jetbrains // {
                idea-ultimate = super.jetbrains.idea-ultimate.overrideAttrs (_: {
                    name = "idea-ultimate-2019.2.8";
                    src = super.fetchurl {
                        url = "https://download.jetbrains.com/idea/ideaIU-2018.2.8-no-jdk.tar.gz";
                        sha256 = "0lah6sb5yaq84c68ak07hsn8k3cjrq4phl4kyiydbhwi09qsc339";
                    };

                });
                jdk = danbstFixedPkgs.jetbrains.jdk.overrideAttrs (oldAttrs: rec {
                    version = "520.11";
                    src = super.fetchurl {
                        url = "https://bintray.com/jetbrains/intellij-jbr/download_file?file_path=jbrsdk-11_0_4-osx-x64-b520.11.tar.gz";
                        sha256 = "0d1qwbssc8ih62rlfxxxcn8i65cjgycdfy1dc1b902j46dqjkq9z";
                    };
                    just-to-retain-a-link-between-garbage-collection-to-patched-nixpkgs = danbstFixedNixpkgs;
                });
            };
          })
      ];
  };
in pkgs.jetbrains.idea-ultimate
15

PR WIP jdk: use `placeholder` for jdk.home by danbst · Pull Request #76571 · NixOS/nixpkgs · GitHub

16

Seems like that PR is wrong. I also found a way to override without nixpkgs patching:

let
  pkgs = import ./nixpkgs {
      config.allowUnfree = true;
      overlays = [
          (self: super: {
            jetbrains = super.jetbrains // {
                idea-ultimate = super.jetbrains.idea-ultimate.overrideAttrs (_: {
                    name = "idea-ultimate-2019.2.8";
                    src = super.fetchurl {
                        url = "https://download.jetbrains.com/idea/ideaIU-2018.2.8-no-jdk.tar.gz";
                        sha256 = "0lah6sb5yaq84c68ak07hsn8k3cjrq4phl4kyiydbhwi09qsc339";
                    };
                });
                jdk = super.jetbrains.jdk.overrideAttrs (oldAttrs: rec {
                    version = "520.11";
                    src = super.fetchurl {
                        url = "https://bintray.com/jetbrains/intellij-jbr/download_file?file_path=jbrsdk-11_0_4-linux-x64-b520.11.tar.gz";
                        sha256 = "0d1qwbssc8ih62rlfxxxcn8i65cjgycdfy1dc1b902j46dqjkq9a";
                    };
                    passthru = oldAttrs.passthru // {
                        home = self.jetbrains.jdk;
                    };
                });
            };
          })
      ];
  };
in pkgs.jetbrains.idea-ultimate

The important part was to override also passthru.home, which had a link to old jdk.

1 Like
17

Thanks so much @danbst!

I assume import ./nixpkgs can be replaced happily with import <nixpkgs> or fetchTarball etc as previously suggested.

Being new to nix, I’d be interested to know how you went about debugging this.

18

@danbst nearly there. Thanks so much for your help!

So I’ve taken what you had into an overlay ~/.config/nixpkgs/overlays/02-jetbrains.nix

self: super:
{
  jetbrains = super.jetbrains // {
    jdk = super.jetbrains.jdk.overrideAttrs (oldAttrs: rec {
      version = "520.11";
      src = super.fetchurl {
        url = "https://bintray.com/jetbrains/intellij-jbr/download_file?file_path=jbrsdk-11_0_4-osx-x64-b520.11.tar.gz";
  	sha256 = "3fe1297133440a9056602d78d7987f9215139165bd7747b3303022a6f5e23834";
      };
      passthru = oldAttrs.passthru // {
        home = self.jetbrains.jdk;
      };
    });
    idea-ultimate = super.jetbrains.idea-ultimate.overrideAttrs (_: {
      name = "idea-ultimate.2019.2.4";
      src = super.fetchurl {
        url = "https://download.jetbrains.com/idea/ideaIU-2019.2.4-no-jbr.tar.gz";
	sha256 = "09mz4dx3zbnqw0vh4iqr8sn2s8mvgr7zvn4k7kqivsiv8f79g90a";
      };
    });
  };
}

This installs happily via nix-env -iA 'nixpkgs.jetbrains.idea-ultimate’.

But when executing it complains it doesn’t know about the JDK_HOME.

ldeck@ldeck ~ % which idea-ultimate
/Users/ldeck/.nix-profile/bin/idea-ultimate

ldeck@ldeck ~ % idea-ultimate
Unable to find any JVMs matching version "(null)".
No Java runtime present, try --request to install.
ERROR: Cannot start IntelliJ IDEA
No JDK found. Please validate either IDEA_JDK, JDK_HOME or JAVA_HOME environment variable points to valid JDK installation.

It looks like the JDK_HOME is meant to be set during the install phase from applications/editors/jetbrains/common.nix.

Any thoughts on what might be missing? Thanks again!

19

Hmm, the executable does have a JDK_HOME…

ldeck@ldeck ~ % cat $(which idea-ultimate)    
#! /nix/store/m7g5jss6wkwx7b8m2x7qg3wdb4x0whw6-bash-4.4-p23/bin/bash -e
export PATH='/nix/store/qz2bg8ga3qnf3pnyiwrbzmpp5h8zsi6h-idea-ultimate.2019.2.4/libexec/idea-ultimate-2019.2.4:/nix/store/bsylyydil2yvxmfr96dvv8xg5h590x2k-jetbrainsjdk-520.11/jdk/Contents/Home/bin:/nix/store/bsylyydil2yvxmfr96dvv8xg5h590x2k-jetbrainsjdk-520.11/bin:/nix/store/2ck9rfwnivsj5lyzr35v3gf9854xm33a-coreutils-8.31/bin:/nix/store/s2ixkwlyqq2h3l6rbrapf2xfxkly7dsc-gnugrep-3.3/bin:/nix/store/jkh8498r42qx0kr61qb7n43lxa694y5y-which-2.21/bin:/nix/store/3fing5pkdda7l7d4h16yq07siqikdjmd-git-2.24.1/bin'${PATH:+':'}$PATH
export LD_LIBRARY_PATH='/nix/store/jvzc6v8sjwpws3kdq01m0im582kbiv47-clang-7.1.0-lib/lib:/nix/store/jd3vmjkfnf38ls52sxvrkv6flcpb3ppl-libsecret-0.19.1/lib:/nix/store/hp90sbwznq1msv327f0lb27imvcvi80h-libnotify-0.7.8/lib'${LD_LIBRARY_PATH:+':'}$LD_LIBRARY_PATH
export JDK_HOME='/nix/store/bsylyydil2yvxmfr96dvv8xg5h590x2k-jetbrainsjdk-520.11'
export IDEA_JDK='/nix/store/bsylyydil2yvxmfr96dvv8xg5h590x2k-jetbrainsjdk-520.11'
export ANDROID_JAVA_HOME='/nix/store/bsylyydil2yvxmfr96dvv8xg5h590x2k-jetbrainsjdk-520.11'
export JAVA_HOME='/nix/store/bsylyydil2yvxmfr96dvv8xg5h590x2k-jetbrainsjdk-520.11'
exec "/nix/store/qz2bg8ga3qnf3pnyiwrbzmpp5h8zsi6h-idea-ultimate.2019.2.4/idea-ultimate.2019.2.4/bin/idea.sh"  "$@"

So I wonder why it thinks it’s not set?

20

The path to JAVA_HOME, JDK_HOME etc is wrongly set.

It needs the suffix path /Contents/Home.

...
export JDK_HOME='/nix/store/bsylyydil2yvxmfr96dvv8xg5h590x2k-jetbrainsjdk-520.11/Contents/Home’
...

Do you require this on linux as well @danbst?