buildPythonPackage throwing dependency-version errors when I don't belive it should be

Hello, me again! Can you believe its only been a few hours since I asked my last embarrassing amatureish question regarding buildPythonPackage-related activities?

I’m continuing on my quest to successfully package Copier (GitHub - copier-org/copier: Library and command-line utility for rendering projects templates.). As I’ve been working my way through its dependencies, I’ve run into two variations of the same issue that’s a bit confusing for me, I’ll try to describe both:

(All dependencies are being provided through propagatedBuildInputs)

ERROR 1 – PYYAML-INCLUDE

There is one dependency called pyyaml-include, which is not available in nixpkgs, so I’m building it myself (also via buildPythonPackage). Copier requires the version to be >=1.2, and the latest version provided is 1.2.post2. I am building the package from master (GitHub - tanbro/pyyaml-include: yaml include other yaml) because the build fails for reasons I don’t understand when i try to use specific tags (like v1.2.post2 or even 1.2).

Copier’s build throws errors saying Could not find version...pyyaml-include<2.0;>=1.2 (from copier) (from versions: none), even though I can see python3.9-pyyaml-include-1.2.post2.drv is successfully built above. I suspect the less-than-standard version-naming scheme is responsible for this particular issue? When I change Copier’s requirement listed in pyproject.toml from "^1.2" to "=1.2.post2", I do not encounter this error. However, of course, I ideally shouldn’t need to edit the actual dependency requirements to get things to build on nix :confused:

ERROR 2 – PLUMBUM

The next dependency is called plumbum, which is actually provided by nixpkgs as python39Packages.plumbum, and is version 1.7.0 in the nixpkgs used (I made sure, in 20.09 its at version 1.6.9). Copier’s version requirement for plumbum is >=1.6.9. As far as I understand, 1.7.0 is semantically greater than 1.6.9. Lo, I am met with the same error: Could not find version...plumbum<2.0.0;>=1.6.9 (from copier) (from versions: none). I was paranoid enough that I tried packaging this one as well like I had ones before, but to no avail. On this time around, my previous shenanigan of setting the requirement in pyproject to "1.7.0" did not work.

I am approximately 75% sure I didn’t misspell anything (that may not seem high, but I’m rarely confident about anything) as I copy-pasted the names of packages from pyproject.toml to flake.nix. I’m hardstuck on this last error, but I’d also very much like to understand the first one as well.

My flake is as follows:

{
  description = "A library for rendering project templates";

  inputs = {
    flake-utils.url = "github:numtide/flake-utils";
    nixpkgs.url = "github:NixOS/nixpkgs";

    #
    # Non-nix-provided dependencies
    #
    poetry-dynamic-versioning = {
      url = "github:mtkennerly/poetry-dynamic-versioning";
      flake = false;
    };
    iteration-utilities = {
      url = "github:MSeifert04/iteration_utilities";
      flake = false;
    };
    jinja2-ansible-filters = {
      # Can't use typical gitlab access method 'cause subgroup :/
      url =
        "git+https://gitlab.com/dreamer-labs/libraries/jinja2-ansible-filters";
      flake = false;
    };
    # python39Packages.jinja2 is at version 3.0.1, we need 3.0.2
    jinja2 = {
      url = "github:pallets/jinja2";
      flake = false;
    };
    dunamai = {
      url = "github:mtkennerly/dunamai";
      flake = false;
    };
    pyyaml-include = {
      url = "github:tanbro/pyyaml-include";
      flake = false;
    };
    plumbum = {
      url = "github:tomerfiliba/plumbum/v1.7.0";
      flake = false;
    };
  };

  outputs = { self, nixpkgs, flake-utils, ... }@inputs:
    let
      pkgs = nixpkgs.legacyPackages."x86_64-linux";
      pyPackages = pkgs.python39Packages;
      pyBuild = pyPackages.buildPythonPackage;
      poetry-dynamic-versioning = pyBuild {
        pname = "poetry-dynamic-versioning";
        version = "0.13.1";
        src = inputs.poetry-dynamic-versioning;
        format = "pyproject";
        propagatedBuildInputs = with pkgs // pyPackages; [ poetry python updatedJinja2 dunamai ];
      };
      iteration-utilities = pyBuild {
        pname = "iteration_utilities";
        version = "0.11.0";
        src = inputs.iteration-utilities;
        doCheck = false; # Dies on setuptoolsCheckPhase, idk why sooo... /shrug
      };
      updatedJinja2 = pyBuild {
        pname = "Jinja2";
        version = "3.0.2";
        src = inputs.jinja2;
        propagatedBuildInputs = with pyPackages; [ pyyaml markupsafe ];
      };
      jinja2-ansible-filters = pyBuild {
        pname = "jinja2-ansible-filters";
        version = "1.3.0";
        src = inputs.jinja2-ansible-filters;
        propagatedBuildInputs = with pyPackages; [ pyyaml updatedJinja2 ];
      };
      dunamai = pyBuild {
        pname = "dunamai";
        version = "1.7.0";
        src = inputs.dunamai;
        format = "pyproject";
        propagatedBuildInputs = with pyPackages; [ poetry ];
      };
      pyyaml-include = pyBuild {
        pname = "pyyaml-include";
        version = "1.2.post2";
        src = inputs.pyyaml-include;
        doCheck = false;
        format = "pyproject";
        propagatedBuildInputs = with pyPackages; [ poetry pyyaml ];
      };
      plumbum170 = pyBuild {
        pname = "plumbum";
        version = "1.7.0";
        src = inputs.plumbum;
        doCheck = false;
        propagatedBuildInputs = with pyPackages; [ poetry ];
      };

      copier = pyBuild rec {
        pname = "copier";
        version = "5.1.0";
        src = ./.;
        format = "pyproject";
        propagatedBuildInputs = with pyPackages; [
          pyyaml-include
          poetry-core
          poetry-dynamic-versioning
          cached-property
          colorama
          importlib-metadata
          iteration-utilities
          pathspec
          updatedJinja2
          jinja2-ansible-filters
          plumbum170
          pytest
          pygments
          questionary
          poetry
          pydantic
          markupsafe
        ];
        meta = with nixpkgs.lib; {
          description = ''
            A library for rendering project templates.
          '';
          homepage = "https://github.com/copier-org/copier";
          license = licenses.mit;
        };
      };
    in {
      # Nixpkgs overlay providing the application
      overlay = (final: prev: { inherit copier; });
    } // (flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = import nixpkgs {
          inherit system;
          overlays = [ self.overlay ];
        };
      in rec {
        apps = { copier = pkgs.copier; };

        defaultApp = apps.copier;
      }));
}

Thought I would try my hand at this, and it was very painful. PR is: copier: init at 7.0.1 by jonringer · Pull Request #145528 · NixOS/nixpkgs · GitHub

I also had to create Able to set version using environment variable · Issue #62 · mtkennerly/poetry-dynamic-versioning · GitHub as copier doesn’t have a mechanism to set the version number outside of vcs.

1 Like

This is because the package wasn’t packaged correctly, the fix would have been: https://github.com/NixOS/nixpkgs/pull/145528/commits/9d07d2b032a8a7a80fae9f33d0773f7e1f88e622. Succinctly, it needed SETUPTOOLS_SCM_PRETEND_VERSION = version; as, the publishing toolchain expected to find version information from vcs, but it wasn’t being communicated, so the “0.0.0” stub was being propagated

1 Like

Ahhhhh very interesting ok, I really thought I was going insane. I guess I thought in my head that the version would be derived from the package’s version, had no idea it referenced anything else.

We could create a hook that sets the version in pyproject.toml and one that can remove sections such as of this plugin.

I attempted to do this in the patch phase, and some plugins like poetry-dynamic-versioning will actually error out as they expected the "0.0.0" stub to be present.