Using debBuild to generate a Python dependency archive

Just a little message to show you how I used debBuild.

At my work we use Debian9 on the servers of production. We need to use an application that uses
Python3.6 (Debian9 uses Python3.5).

So the most efficient solution was for us to build Py36 and use venv for the rest of the packages.

So I used debBuild to build Python36, add the Python dependencies of the application and of course make the deb archive that will be deployed with Ansible.

This command allows me to do all this (the NIX_PATH is changed because there is a prob. with Packages.tgz in the master (see the #)):

NIX_PATH=nixpkgs=https://github.com/apeyroux/nixpkgs/archive/px-up-debian-mirror.tar.gz nix-build release.nix --no-sandbox -A python-mce-deb
[/details]

--no-sandbox is useful to have the network in the VM doing the build.

release.nix:

with import <nixpkgs> {};

let

  vm = vmTools.diskImageFuns.debian9x86_64 {
    extraPackages = [
      "libldap2-dev"
      "libssl-dev"
      "zlib1g-dev"
    ] ++ [
      "ca-certificates"
      "dhcp-client"
      "isc-dhcp-client"
      "libsasl2-dev"
    ];
  };

  python-mce = stdenv.mkDerivation {
    name = "python-mce";
    src = ./.;
  };
  
  vmToolsWithNet = vmTools.override {
    rootModules = vmTools.modulesClosure.rootModules ++ ["virtio_net"];
  };

  version = python36.version;
  
  python-mce-deb = pkgs.releaseTools.debBuild {
    name = "python-mce-deb";
    debName = "python-mce-${version}";
    diskImage = vm;
    vmTools = vmToolsWithNet;
    memSize = 2048;
    src = ./.;
    unpackPhase = ''
    cp ${python-mce.src}/aiomda-requirements.txt aiomda-requirements.txt
    cp ${python-mce.src}/common-requirements.txt common-requirements.txt
    tar -xvf ${python36.src} -C . --strip-components=1
    '';
    debRequires = ["openssl"];
    QEMU_OPTS = "-net nic,netdev=user.0,model=virtio -netdev user,id=user.0";
    doCheck = false;
    doInstallCheck = false;
    doInstallPhase = false;
    # configureFlags = ["--with-ensurepip=install" "--without-pymalloc"];
    configureFlags = ["--with-ensurepip=install"];
    preInstall = ./scripts/pre-install-python.sh;
    installCommand = ./scripts/build-python.sh;
    # installCommand = "${python36.withPackages(p: [p.pip p.setuptools])}/bin/python -m pip install -r ./requirements.txt -t /usr/local/py36";
    meta.description = "Python36-MCE";
    meta.version = version;
  };

in {

  python-mce-deb = python-mce-deb;

}

pre-install-python.sh:

touch /etc/fstab
dhclient

build-python.sh:

echo "==== INSTALL Python ===="
make altinstall
echo "==== INSTALL MCE Reqs ===="
export INSTW_EXCLUDE="${INSTW_EXCLUDE}/homeless-shelter/,"
python3.6m -m pip install --no-cache-dir -r ./common-requirements.txt
python3.6m -m pip install --no-cache-dir -r ./aiomda-requirements.txt

If you have ideas or ways to do things more fun don’t hesitate !

That’s a pretty impure build, installing things with pip during build-time. I imagine its convenient but not really the Nix way!

I totally agree :slight_smile:
Then maybe there are better ideas? Maybe better exploit pypi2nix ? The target is a Debian that can’t be too customized

If you just want a virtual env, you could prefetch the sdists/wheels and let pip use those