Mach-nix: Create python environments quick and easy

Announcements
44

mach-nix 2.0.0 realeased

I’m excited to announce the release of mach-nix 2.0.0! (Changelog)
It comes with several new features which significantly increase the success rate out of the box, plus it gives you some nice tools to fix problems in case there are any. Besides python wheel support and improved nixpkgs support it brings capabilities for overlays and python overrides which makes it composible with any other python overrides or nixpkgs overlays.

At its core, mach-nix now simply takes a requirements.txt and an arbitrary pkgs.python, and returns a set of python overrides which, if applied, make your pkgs.python conform to your requirements.txt.

Using mach-nix is not anymore an ‘in or out’ decision. Since it harmonizes with the nixpkgs-way of building python environments, it can be mixed, extended and modified in the usual way, or even applied ontop of an existing configuration.

The feature I’m most excited about is the concept of Providers which allows you to freely prioritize the origin and buildSystem for your packages on a granular basis.

The following 3 providers are available in version 2.0.0:

  1. nixpkgs: Provides packages directly from nixpkgs without modifying their sources. Has only a few versions available, but has a high success rate and all the nix features, like cudaSupport for tensorflow for example.
  2. sdist: Provides all package versions available from pypi which support setuptools and builds them via nixpkgs overlays wherever possible to resolve external dependencies. It still supports the nixpkgs specific features no matter which package version is selected. But chances are higher for a build to fail than with the nixpkgs provider.
  3. wheel: Provides all linux compatible wheel releases from pypi. Wheels can contain binaries. Mach-nix autopatches them to work on nix. Wheels are super quick to install and work quite reliable. Therefore this provider is preferred by default.

Mach-nix builds environments by mixing packages from all 3 providers. You decide which providers should be preferred for which packages, or which providers shouldn’t be used at all.
The default preferred order of providers is wheel, sdist, nixpkgs.

Providers can be disabled/enabled/preferred like in the following examples:

  • A provider specifier like "wheel,sdist,nixpkgs" means, that the resolver will first try to satisfy the requirements with candidates from the wheel provider. If a resolution is impossible or a package doesn’t provide a wheel release, it falls back to sdist/nixpkgs for a minimal number of packages. In general it will choose as many packages from wheel as possible, then sdist, then nixpkgs.

  • "nixpkgs,sdist" means, that nixpkgs candidates are preferred, but mach-nix falls back to build from source (sdist). wheel is not listed and therefore wheels are disabled.

A full provider config passed to mach-nix looks like this:

{
 # The default for all packages which are not specified explicitly
 _default = "nixpkgs,wheel,sdist"

 # Explicit settings per package
 numpy = "wheel,sdist"
 tensorflow = "wheel"
}

Mach-nix will always satisfy your requirements.txt fully with the configured providers or fail with a ResolutionImpossible error.

If a mach-nix build fails, Most of the times it can be resolved by just switching the provider of a package, which is simple and doesn’t require writing a lot of nix code. For some more complex scenarios, checkout the following examples.

Examples:

1. Tensorflow with SSE/AVX/FMA support

I have a complex set of requirements including tensorflow. I’d like to have tensorflow with the usual nix features enabled like SSE/AVX/FMA which I cannot get from pypi. Therefore i must take tensorflow from nixpkgs. For everything else I keep the default, which means wheels are preferred. This allows for quicker installation of dependencies.

let
  mach-nix = import (builtins.fetchGit {
    url = "https://github.com/DavHau/mach-nix/";
    ref = "2.0.0";
  });
in mach-nix.mkPython {

  requirements = ''
    # bunch of other requirements
    tensorflow
  '';

  providers = {
    # force tensorflow to be taken from nixpkgs
    tensorflow = "nixpkgs"; 
  };
}

This only works if the restrictions in requirements.txt allow for the tensorflow version from nixpkgs. Sadly in this specific case tansorflow needs to be rebuilt and cannot be retrieved from the nix cache. The reason for this is that nixpkgs uses a wrong version of gast for tensorflow. Mach-nix notices that and corrects it which leads to a rebuild. I could include some dont_fixup_nixpkgs option if there is demand for it. But for now I preferred to keep the API simple.

2. Recent Tensorflow quick install

I’d like to install a more recent version of tensorflow which is not available from nixpkgs. Also I hate long build times and therefore I want to install tensorflow via wheel. Usually most wheels work pretty well out of the box, but the tensorflow wheel has an issue which I need to fix with an override.

let
  mach-nix = import (builtins.fetchGit {
    url = "https://github.com/DavHau/mach-nix/";
    ref = "2.0.0";
  });
in mach-nix.mkPython {

  requirements = ''
    # bunch of other requirements
    tensorflow == 2.2.0rc4
  '';

  # no need to specify provider settings since wheel is the default anyways

  # Fix the tensorflow wheel
  overrides_post = [( pythonSelf: pythonSuper: {
    tensorflow = pythonSuper.tensorflow.overridePythonAttrs ( oldAttrs: {
      postInstall = ''
        rm $out/bin/tensorboard
      '';
    });
  })];
}

3. Recent PyTorch with nixpkgs dependencies, overlays, and custom python

I’d like to use a recent version of Pytorch from wheel, but I’d like to build the rest of the requirements from sdist or nixpkgs, since I’ve already written overlays for those packages which I’d like to continue using. Also I require python 3.6

let
  mach-nix = import (builtins.fetchGit {
    url = "https://github.com/DavHau/mach-nix/";
    ref = "2.0.0";
  });
overlays = []; # some very useful overlays
in mach-nix.mkPython rec {

  requirements = ''
    # bunch of other requirements
    torch == 1.5.0
  '';

  providers = {
    # disallow wheels by default
    _default = "nixpkgs,sdist";
    # allow wheels only for torch
    torch = "wheel";
  };

  # Include my own overlay. (Caution! nixpkgs >= 20.03 required for wheel support)
  pkgs = import <nixpkgs> { config = { allowUnfree = true; }; inherit overlays; };

  # Select custom python version (Must be taken from pkgs with the overlay applied)
  python = pkgs.python36;
}

4. Use overrides from poetry2nix

I have a complex requirements.txt which includes imagecodecs. It is available via wheel, but I prefer to build everything from source. This package has complex build dependencies and is not available from nixpkgs. Luckily poetry2nix` overrides make it work. The peotry2nix overrides depend on nixpkgs-unstable.

let
  mach-nix = import (builtins.fetchGit {
    url = "https://github.com/DavHau/mach-nix/";
    ref = "2.0.0";
  });
in mach-nix.mkPython rec {

  requirements = ''
    # bunch of other requirements
    imagecodecs
  '';

  providers = {
    _default = "sdist";
  };

  # Use unstable channel
  pkgs = import <unstable> {};

  # Import overrides from poetry2nix
  # Caution! Use poetry2nix overrides only in `overrides_post`, not `overrides_pre`.
  overrides_post = [
      (
        import (builtins.fetchurl {
          url = "https://raw.githubusercontent.com/nix-community/poetry2nix/1cfaa4084d651d73af137866622e3d0699851008/overrides.nix";
        }) { inherit pkgs; }
      )
    ];
}

Of course those examples are unsafe since they don’t include hashes. But i hope they demonstrate well enough how you can build complex python environments, while using nixpkgs specific features, fixing up packages, speeding up builds, and including your own overlays and overrides.

My plans for upcoming versions:

  • Github Provider: Some complex packages like tensorflow for example don’t publish sdist releases and therefore mach-nix cannot build arbitraty versions of them from source. Therefore the only way to use them with nixpkgs specific build features, is to take exactly the versions from nixpkgs. Nixpkgs builds them from github. If a github provider would be implemented into mach-nix, it could build arbitrary versions of tensorflow from source including all nix features.
  • Draw Dependency Graph: For debugging purposes it would be nice to have a commanline tool to display the resolved graph of a given requirements.txt + provider settings.
  • buildPythonPackage: Fully automatic buildPythonApplication and buildPythonPackage builders, which are able to extract a packages requirements from the package soruce tree.
  • Additional requirement formats: It would be cool to support more requirement formats than just requirements.txt. I have in mind:
    • project.toml for minimal build dependencies.
    • project.toml poetry style
    • Pipfile / Pipfile.lock
    • setup.py / setup.cfg… the usual setuptools stuff
    • … Are there any other important ones?
  • Compression for Database: Add compression for package index and dependency graph to make mach-nix mroe CI friendly.

Let me know if you have any other ideas in mind!

Also if you’d like to contribute or just share your ideas/problems, I would love to see some collaboration on github!

In case any builds fail, please don’t hesitate to open issues on github. This is the only way we can fix these issues or improve docs in case there are some misunderstandings.

If you figure that some package is only working from specific providers, please commit your provider config for that package to This File. Those are the builtin mach-nix defaults. It will help other people to build those packages out of the box.

I did not yet include any default overrides. There has already been invested a lot of energy to fix python packages in nixpkgs (which we are reusing here) and in poetry2nix. I’d like to not open another project with overrides unless it is really necessary. I guess it is in the interest of all 3 projects to agree on a common format for python fixes. Would it be possible to work together on a specification for this? @FRidh @adisbladis Of course with the ultimate goal to use that format also in nixpkgs.

My requirements for fixes are:

  1. Separate treatment for different package formats like sdist, wheel, github/other.
  2. Different treatment for different versions. Fixes could include a version specifier like “>=1.2.0, <2.0.0”. So if my package has version “1.5.0”, I know i need to apply this fix
  3. Separation between handling python inputs and native inputs + fixes. Like already mentioned by you guys.
  4. A defined mapping from pypi package names to nix attribute names.

Unrelated sidenote: I’m a freelancer and looking for some cool projects to collaborate.

8 Likes