Helping alembic find python modules

Help
1

I am creating a docker image for my python project, but alembic always struggles to find my project.

specifically it won’t find matrix-registration in the env.py

import sys

from os.path import abspath, dirname, getcwd

sys.path.insert(0, dirname(dirname(abspath(__file__))))

from matrix_registration import config

 alembic = pkgs.python3Packages.alembic.overrideAttrs (old: {
    propagatedBuildInputs = old.propagatedBuildInputs ++ [
      matrix-registration
    ];
  });
  alembic-cli-wrapper = pkgs.stdenv.mkDerivation {
    name = "alembic-cli-wrapper";
    buildInputs = [ pkgs.makeWrapper ];
    buildCommand = ''
      mkdir -p $out/bin
      makeWrapper ${alembic}/bin/alembic "$out/bin/alembic" \
        --chdir "${matrix-registration}" \
        --add-flags "-x config='${matrix-registration-config}'"
    '';
  };

inspired by:

but that results in:

error: builder for '/nix/store/3z19bmxrdix0dqlfrq32rq1jnllr99gm-alembic-cli-wrapper.drv' failed with exit code 1;
       last 9 log lines:
       >
       > Builder called die: Cannot wrap '/nix/store/fqx3g6112bsbazgjwi6q7sf6vjmx3maf-python3.10-alembic-1.8.1/bin/alembic' because it is not an executable file
       > Backtrace:
       > 7 assertExecutable /nix/store/1mn00xb3j9x364cmki6534g5z77skfc6-hook/nix-support/setup-hook
       > 43 makeShellWrapper /nix/store/1mn00xb3j9x364cmki6534g5z77skfc6-hook/nix-support/setup-hook
       > 36 makeWrapper /nix/store/1mn00xb3j9x364cmki6534g5z77skfc6-hook/nix-support/setup-hook
       > 1366 genericBuild /nix/store/kmfaajdpyyyg319vfqni5jm9wkxjmf73-stdenv-linux/setup
       > 2 main /nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh
       >
       For full logs, run 'nix log /nix/store/3z19bmxrdix0dqlfrq32rq1jnllr99gm-alembic-cli-wrapper.drv'.
error: 1 dependencies of derivation '/nix/store/f82cfvy90vrqr4jz0hc0fkp5gfv3ypqq-matrix-registration.drv' failed to build
error: 1 dependencies of derivation '/nix/store/672nk6pm3iknyjsw0fc26i474h2r1hah-docker-layer-matrix-registration.drv' failed to build
error: 1 dependencies of derivation '/nix/store/zyzqpq8lag7hmb3dzll0m3cwm31v856p-docker-image-matrix-registration.tar.gz.drv' failed to build

/nix/store/fqx3g6112bsbazgjwi6q7sf6vjmx3maf-python3.10-alembic-1.8.1/
├── lib
│   └── python3.10
│       └── site-packages
└── nix-support
    └── propagated-build-inputs
2

That looks like matrix-registration already contains alembic and pip is not installing it again.

You probably want to create a python env with all packages you need python3.withPackages(ps: with ps; [ alembic matrix-registration ]) and then create the wrapper from that or inside of that.

1 Like
3

thank you! that is definitely the right direction, but I am really unsure how to exactly do this?

let
  matrix-registration = import ./shell.nix { inherit pkgs; };
  matrix-registration-config = "/data/config.yaml";

  python = let
    packageOverrides = self: super: {
      inherit matrix-registration;
      alembic = super.alembic.overridePythonAttrs(old: rec {
        makeWrapperArgs = [
          "--chdir '${matrix-registration}'"
          "--add-flags \"-x config='${matrix-registration-config}'\""
        ];
      });
    };
  in pkgs.python3.override {inherit packageOverrides; self = python;};

 python-packages = ps: with ps; [
    alembic
    # matrix-registration # adding both leads to a collision since matrix-registration ofc already has alembic as an input
  ];

in pkgs.dockerTools.buildImage {
  name = "matrix-registration";
  # ...
  copyToRoot = python.withPackages python-packages;
  # ...
}

I can build a python environment like this where alembic is correctly wrapped, but matrix-registration is not in the environment.

Or I create an environment with matrix-registration, but I do not know how to wrap alembic now, since then matrix-registration grabs alembic from the pkgs it is called with.

matrix-registration = import ./shell.nix { inherit pkgs; };
4

Another idea I had was just overwriting the pkgs I call my python package with:

 matrix-registration-config = "/data/config.yaml";
 matrix-registration = import ./shell.nix { pkgs = newpkgs; };

 newpkgs = import pkgs.path { overlays = [ (self: super: {
   python3 = let
     packageOverrides = python-self: python-super: {
       alembic = python-super.alembic.overridePythonAttrs(old: rec {
         makeWrapperArgs = [
           # "--chdir '${matrix-registration}'"
           "--add-flags \"-x config='${matrix-registration-config}'\""
         ];
     });
     };
   in super.python3.override {inherit packageOverrides;};
 } ) ]; };

This does creates a matrix-registration with a wrapped alembic, but I ofc can’t refer to matrix-registation within it’s own input since this leads to infinite recursion.

nix-build docker.nix 
error: infinite recursion encountered

       at /nix/store/02ghmvdvispvni7cynknqhwysx5qlb9y-nixos-22.11/nixos/pkgs/stdenv/generic/make-derivation.nix:321:7:

          320|       depsHostHostPropagated      = lib.elemAt (lib.elemAt propagatedDependencies 1) 0;
          321|       propagatedBuildInputs       = lib.elemAt (lib.elemAt propagatedDependencies 1) 1;
             |       ^
          322|       depsTargetTargetPropagated  = lib.elemAt (lib.elemAt propagatedDependencies 2) 0;
(use '--show-trace' to show detailed location information)
5

I was overthinking this. I ended up just removing alembic from my shell.nix since it’s not really necessary to run matrix-registration (though I do would like it in my alembic in my nix-shell dev enviroment, but that’s for another day ^^)

That way I can easily get both wrapped in the same environment:

let
  matrix-registration-config = "/data/config.yaml";

  python3 = let
     packageOverrides = self: super: rec {
      alembic = super.alembic.overridePythonAttrs(old: rec {
        makeWrapperArgs = [
          "--chdir '${matrix-registration}'"
          "--add-flags \"-x config='${matrix-registration-config}'\""
        ];
      });
      matrix-registration = (import ./shell.nix { inherit pkgs; }).overridePythonAttrs(old: rec {
        makeWrapperArgs = [
          "--add-flags \"--config-path='${matrix-registration-config}'\""
        ];
      });
    };
  in pkgs.python3.override {
    inherit packageOverrides;
    self = python3;
  };

  python-packages = ps: with ps; [
    ps.matrix-registration
    ps.alembic
  ];

in pkgs.dockerTools.buildImage {
  name = "matrix-registration";
  tag = tag;
  created = "now";

  copyToRoot = python3.withPackages python-packages;
  # ...