Packaging a Python website

Help
1

Hello,

I’m trying to package a Django website to get the following:

  1. A Python environment with the dependencies (and the Django website modules) installed
  2. A custom binary manage-myproject, which would set some custom environment variables, and then run the django-admin binary, provided by the Django Python package
  3. An additional Python package, gunicorn, installed in the same Python environment (so that it can run the site with all its dependencies)

My project uses Poetry, and I was able to use poetry2nix to create a Python environment with all the dependencies installed:

(poetry2nix.mkPoetryApplication {
    projectDir = ./.;
    configurePhase = ''
      ln -s ${nodeDependencies} ./blog/staticfiles
    '';
  }).dependencyEnv

Now to get point 2 to work, I tried the following:

let
  app = poetry2nix.mkPoetryApplication {
    projectDir = ./.;
    configurePhase = ''
      ln -s ${nodeDependencies} ./blog/staticfiles
    '';
  };

  manageScript = ''
    #!${pkgs.bash}/bin/bash
    export DJANGO_SETTINGS_MODULE=myproject.config.settings.base
    ${app.python}/bin/django-admin $@
  '';

  managePy = pkgs.runCommand "manage-myproject" {
  } ''
    mkdir -p $out/bin
    echo -e '${manageScript}' > $out/bin/managepy-myproject
    chmod +x $out/bin/managepy-myproject
  '';

  pythonWithApp = pkgs.symlinkJoin {
    name = "myproject";
    paths = [ managePy app.dependencyEnv ];
  };
in pythonWithApp

Which correctly adds a bin/managepy-myproject file, but it seems it’s not pointing to the correct Python environment, because the resulting ${app.python}/bin/django-admin file doesn’t exist (and indeed, the path of the Python interpreter referenced in the file is a Python environment without any dependency installed).

Anyway, trying to get point 3 to work, I tried using the following to get the gunicorn Python application added to the environment:

  pythonWithApp = pkgs.symlinkJoin {
    name = "myproject";
    paths = [
      managePy
      (app.python.withPackages (ps: [ ps.gunicorn app.dependencyEnv ]))
    ];
  };

But this resulted in the other Python applications (such as django-admin) not being included anymore in the Python environment.

So my questions are:

  1. How can I properly reference the Python interpreter from the Python environment that has all the other packages installed?
  2. How can I add a Python application to an existing Python environment, while keeping all other applications?
  3. How can I get rid of unwanted binaries in bin? I tried adding my package to environment.systemPackages and all of these packages were then in the PATH (I would just like to have managepy-myproject):
❯ ls result/bin/
2to3        django-admin     idle3    pydoc3    python-config   python3.8         unidecode
2to3-3.8    django-admin.py  idle3.8  pydoc3.8  python3         python3.8-config  wagtail
chardetect  idle             pydoc    python    python3-config  sqlformat

Any hint would be much appreciated. Thanks!

2

Did you find a solution to this?

3

Kind of! I ended up creating a NixOS module I’m now using in production. It still lacks some docs but you might find what you’re looking for in the code.

Hope this helps!

1 Like
4

Uh wow quite the setup.
Do you have maybe an example package?
I’m looking more for an example on how to package a Django package with multiple apps.

5

Sorry I don’t have a public packaged Django site I could show you, but I’m basically using a poetry2nix setup:

{ poetry2nix, mkShell, python3Packages, stdenv, python39, myproject }: rec {
  server = poetry2nix.mkPoetryApplication {
    projectDir = ./.;
    python = python39;
  };

  collectedStatic = stdenv.mkDerivation {
    pname = "${server.pname}-static";
    version = server.version;
    src = ./.;
    buildPhase = ''
      export STATIC_ROOT=$out
      export DJANGO_SETTINGS_MODULE=myproject.config.settings.base
      export MEDIA_ROOT=/dev/null
      export SECRET_KEY=dummy
      export DATABASE_URL=sqlite://:memory:
      export COLLECTSTATIC_ROOT=${myproject.frontend.static}
      ${server.dependencyEnv}/bin/django-admin collectstatic --noinput
    '';
    phases = [ "buildPhase" ];
  };

  shell = mkShell {
    packages = [
      (poetry2nix.mkPoetryEnv {
        projectDir = ./.;
        editablePackageSources = {
          myproject = ./myproject;
          blog = ./blog;
        };
        overrides = poetryOverrides;
      })
      python3Packages.poetry
    ];
  };
}

I then use this with my django.nix NixOS module like so:

django.sites.myproject = {
  package = pkgs.myproject.server;
  staticFilesPackage = pkgs.myproject.collectedStatic;
  hostname = "myproject.com";
  wsgiModule = "myproject.config.wsgi";
  settingsModule = "myproject.config.settings.base";
};

Note you might not need all the collectstatic stuff if you’re hosting your static on a CDN.

1 Like
6

Thank you for the snippets!
For some reason only the main project directory gets packaged in my project.
Does you repo structure look similar to this?

7

When building your project I’ve noticed all your code ends up in an src package:

❯ ll result/lib/python3.10/site-packages/
dr-xr-xr-x - root  1 Jan  1970 network_inventory-0.1.0.dist-info
dr-xr-xr-x - root  1 Jan  1970 src

This is not what you want, since it would then mean your code should import packages using from src.backups import models for example, instead of from backups import models. You have 2 choices here:

The first is to keep your project structure as-is, and adapt your pyproject.toml file to list all the packages exposed by your project:

[tool.poetry]
name = "network_inventory"
version = "0.1.0"
description = ""
authors = ["Andreas Zweili <andreas@zweili.ch>"]
license = "GPLv3"
packages = [
  { include = "backups", from = "src" },
  { include = "computers", from = "src" },
  { include = "core", from = "src" },
  # include your other packages here…
]

The second is to change your directory structure to put everything in a single package under the src/ directory, like so:

src/
└ network_inventory/
  ├── __init__.py
  ├── backups/
  │   └── __init__.py
  ├── computers/
  │   └── __init__.py
  ├── …/

And update your pyproject.toml file to:

packages = [
  { include = "network_inventory", from = "src" }
]

Option 2 is usually what I do, since it prevents clashes with dependencies that might have the same name (eg. users or core are quite common package names that could conflict).

Hope this helps.

1 Like
8

Just for the record.
I unfortunately wasn’t able to get the second way working.
I managed to get the docker build working, however in a very ugly and hacky way but at least it is working.
For the moment it has to be good enough, at least it doesn’t contain all the development dependencies anymore.