Python3.11 & TensorFlow >=2.12 how to?

Help
1

Hi,

I am trying to install the above-mentioned versions of python and tensorflow.

My try:

let
  # Unstable packages
  unstableTarball = fetchTarball https://github.com/NixOS/nixpkgs/archive/nixos-unstable.tar.gz;

  # Custom Python package with all the (Python) imports I need
  my-python-packages = python-packages: with python-packages; [
    matplotlib
    numpy
    openpyxl
    pandas
    scipy
    tensorflow
    keras
  ]; 
  python-with-my-packages = pkgs.unstable.python311.withPackages my-python-packages;
in
{
  environment.systemPackages = with pkgs; [
    python-with-my-packages
  ];
}

The error I get:

building the system configuration...
error: tensorboard-2.11.0 not supported for interpreter python3.11
(use '--show-trace' to show detailed location information)

Any help I may get to get this to work would be greatly appreciated.
Thank you in advance :slight_smile:
Davide

2

It seems that this line:

Was added in Python updates 2022-03-02 by mweinelt · Pull Request #162538 · NixOS/nixpkgs · GitHub . I wonder why was that. In anycase, it ought to get an update to 2.13.0, and if you test that it works with python 3.11, I think you can write a PR to remove that condition (along with the update) - Since Python 3.6 packages are not built by default anymore in anycase.

3

Sorry for my ignorance, but how should I proceed? Unfortunately I have no experience in writing custom derivations (or contributing to official ones)

4

I hope you have experience writing pull requests on GitHub. The topic of contributing to Nixpkgs is covered in it’s manual:

I’d recommend using a tool like nix-update python3.pkgs.tensorboard --commit to update the package. When inside nixpkgs directory, run nix build -Lf. python3.pkgs.tensorboard to test your changes. If you want to spawn a python interpreter with the updated tensorboard, I’d use - again inside nixpkgs directory:

$ nix shell --impure --expr 'with builtins.getFlake "'$PWD'"; with legacyPackages.${builtins.currentSystem}; python3.withPackages(ps: [ps.tensorboard])'
$ python

Or:

$ nix-shell -I nixpkgs=. -p 'python3.withPackages(ps:[ps.tensorboard])'
(ins)[nix-shell:~/nixpkgs]$ python
5

Thank you. I will try with tensorflow 2.13 on python312 and fix the dependancies whenever I encounter a problem. I will write again if successful (or if stuck)

6

…and indeed here I am, stuck!
I tried to build a test system configuration using test packages, as follows:

  1. Fork nixpkgs, and clone the fork on my hard drive
  2. Modify my configuration.nix:
{ config, pkgs, ... }:
let
  localNixpkgs = ../../nixpkgs;
  my-python-packages = python-packages: with python-packages; [
    tensorflow
    keras
  ]; 
  python-with-my-packages = pkgs.local.python3.withPackages my-python-packages;
in
{
  environment.systemPackages = with pkgs; [
    python-with-my-packages
  ];
}
  1. Modify my copy of tensorboard/default.nix (update version & hash, remove disabled version):
buildPythonPackage rec {
  pname = "tensorboard";
  version = "2.13.0";
  format = "wheel";

  src = fetchPypi {
    inherit pname version format;
    dist = "py3";
    python = "py3";
    hash = "sha256-q2mWHr3b3cg/X6L/kjNXK9rVuIN3jDXk/pS/F5i9hIE=";
  };
# ...
}
  1. Modify my copy of tensorboard-data-server/default.nix because required by tensorboard (update version and hash):
buildPythonPackage rec {
  pname = "tensorboard-data-server";
  version = "0.7.1";
  format = "wheel";
  disabled = pythonOlder "3.6";

  src = fetchPypi {
    pname = "tensorboard_data_server";
    inherit version format;
    dist = "py3";
    python = "py3";
    hash = "sha256-mTi9OfUEF5ezOSEGb7oOqwOg3RDRiHoF5irliEGtTD8=";
  };
# ...
}
  1. Update version in tensorflow/default.nix:
  version = "2.13.0";
  1. sudo nixos-rebuild switch

Spends hours compiling; then the output puzzles me:

...
adding 'tensorflow-2.11.1.dist-info/LICENSE'
adding 'tensorflow-2.11.1.dist-info/METADATA'
adding 'tensorflow-2.11.1.dist-info/WHEEL'
adding 'tensorflow-2.11.1.dist-info/entry_points.txt'
adding 'tensorflow-2.11.1.dist-info/top_level.txt'
adding 'tensorflow-2.11.1.dist-info/RECORD'
removing build/bdist.linux-x86_64/wheel
Finished executing setuptoolsBuildPhase
installing
Executing pipInstallPhase
/build/tensorflow-2.13.0-python/dist /build/tensorflow-2.13.0-python
Processing ./tensorflow-2.11.1-cp310-cp310-linux_x86_64.whl
Requirement already satisfied: wrapt>=1.11.0 in /nix/store/flmr2b7jpaj69npah8z1m6h34klff3b5-python3.10-wrapt-1.14.1/lib/python3.10/site-packages (from tensorflow==2.11.1) (1.14.1)
Requirement already satisfied: tensorflow-estimator<2.12,>=2.11.0 in /nix/store/xx5q9rz53gc3xbsqkxz4wbm5pqzjz4my-python3.10-tensorflow-estimator-2.11.0/lib/python3.10/site-packages (from tensorflow==2.11.1) (2.11.0)
Requirement already satisfied: google-pasta>=0.1.1 in /nix/store/mnq9680s0sxm295ypmd38dli7frjknbd-python3.10-google-pasta-0.2.0/lib/python3.10/site-packages (from tensorflow==2.11.1) (0.2.0)
Requirement already satisfied: absl-py>=1.0.0 in /nix/store/y8vd079kgc4gqk2pl0fqcklp4qf2f72w-python3.10-absl-py-1.4.0/lib/python3.10/site-packages (from tensorflow==2.11.1) (1.4.0)
Requirement already satisfied: opt-einsum>=2.3.2 in /nix/store/7x6ysnnb9x2mraiz06yxsm3l6md052ry-python3.10-opt-einsum-3.3.0/lib/python3.10/site-packages (from tensorflow==2.11.1) (3.3.0)
Requirement already satisfied: packaging in /nix/store/c5fh1flbs76jpgmvzz96xa26c3fwsq2s-python3.10-packaging-23.0/lib/python3.10/site-packages (from tensorflow==2.11.1) (23.0)
Requirement already satisfied: h5py>=2.9.0 in /nix/store/yqm1lmc46hc363jf1xrznwal1gl06zqi-python3.10-h5py-3.8.0/lib/python3.10/site-packages (from tensorflow==2.11.1) (3.8.0)
Requirement already satisfied: termcolor>=1.1.0 in /nix/store/x6f5hkz68csv7wrvbzwwhysmgmxd273c-python3.10-termcolor-2.2.0/lib/python3.10/site-packages (from tensorflow==2.11.1) (2.2.0)
Requirement already satisfied: astunparse>=1.6.0 in /nix/store/iiafls325vzqqcavrhl22hfx753fkqfc-python3.10-astunparse-1.6.3/lib/python3.10/site-packages (from tensorflow==2.11.1) (1.6.3)
ERROR: Could not find a version that satisfies the requirement tensorboard<2.12,>=2.11 (from tensorflow) (from versions: none)
ERROR: No matching distribution found for tensorboard<2.12,>=2.11

error: builder for '/nix/store/639s8lk10dvfspks4qz68ja9c5ffaw04-python3.10-tensorflow-2.13.0.drv' failed with exit code 1
error: 1 dependencies of derivation '/nix/store/2wk68kh1k1hrvs843fnsdqysjl8qd663-python3-3.10.12-env.drv' failed to build
error: 1 dependencies of derivation '/nix/store/hsy59cqc7k6jjl8zqr6ca14z9nsv6mmz-system-path.drv' failed to build
error: 1 dependencies of derivation '/nix/store/rdppl78k3fzxvjs0ikmzmhqal43immnb-nixos-system-nixos-22.11.1069.cbe419ed4c8.drv' failed to build

It seems to build tensorflow-2.13.0, but then it takes stuff from the previous tensorflow-2.11.0 and then complains because the latter is incompatible with the new tensorboard version?

7

Updated version and hash? If you could also fork your nixpkgs repository on GitHub, and push your branch there and link to it here, it would be easier to help you. You could also open a draft PR and I can help you there.

8

There are only three hash codes, but only under rules_cc_darwin_patched (I am on a Lenovo laptop).

Thank you for your help: my forked repository.

Should not I get my fork to work properly before opening a pull request?

9

You can open a draft pull request, that way the maintainers don’t get notified yet but it’s clear that someone is working on this, which can be helpful if others run into the same issue :slight_smile:

10

what happens if I modify the code after opening the pull request? Shall I just push to my fork and it is automatically included in the pull request?

11

Yes exactly, the PR is updated immediately when you push to your fork.

1 Like
12

I see - Ok, I have just opened a draft pull request. Thank you for your suggestions