Mach-nix: Create python environments quick and easy

Trick nix into caching

  • Calculate hashes from hashes?
  • In other words: unique name-version.

— Not thought into detail, but wouldn’t that work?

Released 2.4.0 (20 Sep 2020)

Global conditional overrides, simple overrides for buildPythonPackage, improved metadata extraction, fix wheel selection

TL;DR;
Mach-nix now has a global override system (similar to poetry2nix). Please commit your fixes to this file. The format is designed to be human readible, conditional and reusable for other projects.

Features

  • Global conditional overrides: Similar to the overrides from poetry2nix, this allows users to upstream their ‘fixes’ for python packages. Though, a special format is used here which is optimized for human readability and allows to define a condition for each fix. Therefore fixes are applied on a granular basis depending on the metadata of each package like its version, python version, or provider. This format is designed in a way, so it could easily be reused by projects other than mach-nix. Please contribute your fixes to ./mach_nix/fixes.nix
  • Simplified overrides are now also available for buildPythonPackage (underscore argument)
  • Inherit passthru from nixpkgs: Reduces risk of missing attributes like numpy.blas.
  • Allow passing a string to the python argument of mkPython: Values like, for example, "python38" are now accepted in which case pkgs.python38 will be used. The intention is to reduce the risk of accidentally mixing multiple nixpkgs versions.
  • Improved error handling while extracting metadata from python sources in buildPythonPackage.

Fixes

  • Selecting extras when using buildPythonPackage didn’t have any effect
  • The passthru argument for buildPythonPackage was ignored
  • The propagatedBuildInputs argument for buildPythonPackage was ignored
  • Wheels with multiple python versions in their filename like PyQt5-...-cp35.cp36.cp37.cp38-...whl were not selected correctly.

Package Fixes:

  • tensorflow: collision related to tensorboard
  • orange3: broken .so file caused by fixupPhase (probably due to shrinking)
  • ldap0: add misssing build inputs.
3 Likes

Sorry, some major stuff was broken in 2.4.0. Here a quick bugfix release:

2.4.1 (21 Sep 2020)

bugfixes

Fixes

  • extra_pkgs was broken: Packages didn’t end up in final environment
  • null value error when inheriting passthru for disabled packages
  • Wrong provider detected for sdist packages in fixes.nix
  • overrides from fixes.nix didn’t apply for buildPythonPackage

Package Fixes

  • pip: allow from sdist provider
  • pip: remove reproducible.patch for versions < 20.0

In case maintainers of other tools like to use the fixes.nix overrides from mach-nix. The following function can be used to transform the custom format to conventional nixpkgs python overrides:
(a package’s provider is assumed to be “nixpkgs” if a package doesn’t define passthru.provider)

function
with pkgs.lib;
let
  fixes = import ./fixes.nix { inherit pkgs; };
  meets_cond = oa: condition:
    let
      provider = if hasAttr "provider" oa.passthru then oa.passthru.provider else "nixpkgs";
    in
      condition { prov = provider; ver = oa.version; pyver = oa.pythonModule.version; };
  combine = pname: key: val1: val2:
    if isList val2 then val1 ++ val2
    else if isAttrs val2 then val1 // val2
    else if isString val2 then val1 + val2
    else throw "_.${pname}.${key}.add only accepts list or attrs or string.";
in
  flatten (flatten (
    mapAttrsToList (pkg: p_fixes:
      mapAttrsToList (fix: keys: pySelf: pySuper:
        let cond = if hasAttr "_cond" keys then keys._cond else ({prov, ver, pyver}: true); in
        if ! hasAttr "${pkg}" pySuper then {} else
        {
          "${pkg}" = pySuper."${pkg}".overrideAttrs (oa:
            mapAttrs (key: val:
              trace "\napplying fix '${fix}' for ${pkg}:${oa.version}\n" (
                if isAttrs val && hasAttr "add" val then
                  combine pkg key oa."${key}" val.add
                else if isAttrs val && hasAttr "mod" val && isFunction val.mod then
                  val.mod oa."${key}"
                else
                  val
              )
            ) (filterAttrs (k: v: k != "_cond" && meets_cond oa cond) keys)
          );
        }
      ) p_fixes
    ) fixes
  ));
1 Like

mmm, I’m currently trying to use mach-nix to install Plone and it’s dependencies, but it’s running since 50 minutes and currently is consuming 50.8G of resident memory… it printed the dependency tree but since then nothing more… any way to know more about what is it doing?

aargh… it got just killed by the OOM killer!
I know that the plone dependency tree is huge, but I really hoped that with some amount of memory and time, it could complete the env task

That doesn’t look too good :wink: I opened an issue for this matter here: https://github.com/DavHau/mach-nix/issues/158
It would be great if you could comment on the issue and post the exact nix expression or command you used for that build. I wasn’t able to reproduce it.

Released 3.0.0 (14 Oct 2020)

flakes pypi gateway, R support, new output formats, more packages for python 3.5/3.6, improved providers nixpkgs/wheel

IMPORTANT NOTICE

The UI has been reworked. It is backward compatible with a few exceptions. Most importantly, when importing mach-nix, an attribute set must be passed. It can be empty. Example:

let
  mach-nix = import (builtins.fetchGit {
    url = "https://github.com/DavHau/mach-nix/";
    ref = "refs/tags/3.0.0";
  }) {
    # optionally bring your own nixpkgs
    # pkgs = import <nixpkgs> {};

    # or specify the python version
    # python = "python38";
  };
in
...

Features

  • Flakes gateway to pypi. Get a nix shell with arbitrary python packages. Example:

    nix develop github:davhau/mach-nix#shellWith.requests.tensorflow.aiohttp

  • or a docker image
    nix build github:davhau/mach-nix#dockerImageWith.package1.package2 ...

  • or a python derivation
    nix build github:davhau/mach-nix#with.package1.package2 ...

  • New output formats:

    • mkDockerImage -> produces layered docker image containing a python environment
    • mkNixpkgs -> returns nixpkgs which is conform to the given requirements
    • mkOverlay -> returns an overlay function to make nixpkgs conform to the given requirements
    • mkPythonOverrides -> produces pythonOverrides to make python conform to the given requirements.
  • New functions fetchPypiSdist and fetchPypiWheel. Example:

    mach-nix.buildPythonPackge {
      src = mach-nix.fetchPypiSdist "requests" "2.24.0"
    };
    
  • When using the mach-nix cmdline tool, the nixpkgs channel can now be picked via:

    mach-nix env ./env -r requirements.txt --nixpkgs nixos-20.09
    
  • R support (experimental): R packages can be passed via packagesExtra. Mach-nix will setup rpy2 accordingly. See usage example.

  • Non-python packages can be passed via packagesExtra to include them into the environment.

Improvements

  • rework the logic for inheriting dependencies from nixpkgs
  • fixes.nix: allow alternative mod function signature with more arguments:
    key-to-override.mod = pySelf: oldAttrs: oldVal: ...;
  • allow derivations passed as src argument to buildPythonPackage
  • stop inheriting attribute names from nixpkgs, instead use normalized package names
  • rework the public API of mach-nix (largely downwards compatible)
  • add example on how to build aarch64 image containing a mach-nix env
  • tests are now enabled/disabled via global override which is more reliable
  • raise error if python version of any package in packagesExtra doesn’t match to one of the environment

Fixes

  • nixpkgs packages with identical versions swallowed
  • pname/version null in buildPythonPackage
  • update dependency extractor to use “LANG=C.utf8” (increases available packages for python 3.5 and 3.6)
  • wheel provider picked wheels incompatible to python version
  • unwanted python buildInput inheritance when overriding nixpkgs
  • properly parse setup/install_requires if they are strings instead of lists

Package Fixes

  • rpy2: sdist: remove conflicting patch for versions newer than 3.2.6
  • pytorch from nixpkgs was not detected as torch
  • pyqt5: fix for providers nixpkgs and wheel
  • httpx: remove patches
5 Likes

@DavHau my requirements has some packages like h5py and theano. I’m using Mach-Nix 3.0.0 and When I activate my shell.nix it show this message:

Multiple nixkgs attributes found for h5py-2.10.0: ['h5py', 'h5py-mpi']
Picking 'h5py' as base attribute name.
Multiple nixkgs attributes found for python-dateutil-2.8.1: ['dateutil', 'python-dateutil']
Picking 'dateutil' as base attribute name.
Multiple nixkgs attributes found for theano-1.0.5: ['Theano', 'TheanoWithCuda', 'TheanoWithoutCuda']
Picking 'Theano' as base attribute name.

It says there are more than one package to that requirement. How can I force Mach-Nix to pick TheanoWithCuda, for instance?

Good question!
First up, If you use the wheel dependency provider (which is the default), then those attribute names have absolutely no effect.

Only if you use the sdist or nixpkgs provider, mach-nix will inherit the capabilities of the nixpkgs package.

Currently there is no support for selecting those variants.
It automatically prioritizes the nixpkgs candidate with the version most similar to the selected one. If multiple nixpkgs candidates have the same version, the one with the shortest attribute name will be picked.

Manually selecting these variants sounds like a good idea. I will try to add this feature soon.

In the meantime you could, for example, include an override via overridesPre which removes the normal Theano from the attribute set, or marks it broken or disabled. Or set’s its source/pname/version to some horrendous value so mach-nix has no chance to identify the package properly.

Of course if you do that, you will probably also break TheanoWithCuda since it inherits from Theano. Therefore you’d need to fix the earlier messed up values again with another override for TheanoWithCuda.

1 Like

Released 3.0.1 (21 Oct 2020)

bugfixes, return missing packages

Fixes

  • Some sdist packages were missing from the dependency DB due to a corrupt index in the SQL DB used by the crawler.
  • When automatically fixing circular deps, removed deps could trigger a No matching distribution found error in higher level parent packages. Now --no-dependencies is set recursively for all parents of removed deps.
  • Mapping out the resulting dependency DAG to a tree for printing could exhaust the systems resources, due to complexity. Now, when printing dependencies, sub-trees are trimmed and marked via (…) if they have already been printed earlier.

Improvements

  • optimized autoPatchelfHook for faster processing of large wheel packages (see upstream PR)
  • networkx is now used for dealing with some graph related problems
2 Likes

Released 3.0.2 (27 Oct 2020)

bugfixes

Fixes

  • fixed “\u characters in JSON strings are currently not supported” error, triggered by some packages using unicode characters in their file names
  • mach-nix cmdline tool didn’t use specified python version
  • wheel provider was broken for MacOS resulting in 0 available packages
  • several issues triggering infinite recursions

I’m excited to announce that conda support is just around the corner. It will probably be released in form of a beta version very soon.

2 Likes

Quick call for your opinions/recommendations.

In the near future I’d like to concentrate on improving the general UX for mach-nix for non nix users.
I believe nix has the potential to attract many more python users if we make it as user friendly as pip and conda.

One important part which I’m not quite happy about yet is the installation itself.

Currently mach-nix requires nix to be installed via:
curl -L https://nixos.org/nix/install | sh

I see a problem with this because it:

  • seems scary
  • asks for super user rights
  • makes a lot of assumptions on the environment and often crashes (try executing inside ubuntu docker etc…)

It would be optimal to package nix itself as a pypi package and/or conda package.
I’ve seen floating around some instructions on how to install nix without sudo, so i guess it should be possible.

Is there any technical obstacle that makes this idea infeasible? I’m happy for any material or idea that helps me realizing this.

2 Likes

You could create a statically linked Nix and distribute that via say PyPI. Still, you will need /nix/store and thus super user. If your users are fine with working in a subshell, you could use a chroot store instead $ nix shell nixpkgs#python3 --store ~/nix. Note this seems to be broken with nix 2.4pre20201102_550e11f.

Thanks. It seems like chroot won’t work on multiple distros by default since it requires userspaces.
An alternative seems to be proot which doesn’t require any privileges.

Sadly nix 3.x fails when building statically via pkgs.pkgsStatic.nixFlakes.
I guess I cannot rely on the pkgsStatic set, since it is not evaluated by hydra.

Would it be possible to add a static nix (pkgs.nixStatic) to nixpkgs, so any breakage becomes a blocker?

An alternative to not rely on static builds would be nix-bundle. It currently also relies on chroot which I’d have to replace with proot first (WIP PR).

…but I feel like using a static build might be the better solution since it is slimmer and contains less stuff. Also I’m not sure if the licenses of the tools contained in nix-bundle will make distribution more complicated.

I’m aware that the nix build sandbox also cannot be used if userspaces are not available, but I assume for most python related things this shouldn’t matter too much.

Users using the portable nix solution could be warned with a suggestion to do an actual installation of nix before going to production.

To clarify, the reason I mentioned both chroot and static is that

  • static is easier to distribute than the default shared linked;
  • chroot/proot/namespaces is to avoid needing a rebuild when putting the store elsewhere than /nix/store, thereby avoiding super user rights.

It would have to be a blocker in nixos/nix then first.

Released conda-beta (21 Sep 2020)

Conda provider, Conda parser

TL;DR;

This version introduces a new provider for conda packages and the capability of parsing conda’s environment.yml file. Those are independent features, meaning you can access packages from conda without using the conda environment.yml format, or you can build from an environment.yml file while taking packages from pypi or nixpkgs.

The default provider order is updated to: conda,wheel,sdist,nixpkgs
The provider conda is a shorthand combining two conda channels conda/main,conda/r.

Other conda channels are also available. Just use the channel name as a provider, like:
conda-forge,wheel,nixpkgs

If a conda channel is not yet known by mach-nix, add it via condaChannelsExtra

System dependencies of conda packages are satisfied via the anaconda repo. Those system libraries will take up additional closure space. Therefore the conda provider might not be the optimal choice for container scenarios.

Motivation

Obviously it comes in handy being able to transfer conda based environments to nix. But also people, who never used conda before, might benefit from an improved user experience thanks to this update.

I’d like to mention a few important advantages of the conda ecosystem, which make me believe, the conda provider should become the default provider for future stable versions of mach-nix.

Convertability

Compared to pypi, conda packages/environments seem to be much easier to convert to nix. All meta data of packages (dependencies, etc.) are provided by anaconda via a single file per each repo called repodata.json. This file contains all information necessary to convert conda requirements to nix expressions.
Another important factor is, that conda packages declare their non-python dependencies.

Build variants

Of course nothing can compete with nixpkgs in terms of flexibility, but python packages from nixpkgs create a major problem in conjunction with mach-nix. Since mach-nix always changes some dependencies/attributes, every package must be re-built locally which is not much fun for larger packages.
Like nixpkgs, conda provides different build variants for some packages and those are cheap to install.

Non-python dependencies

Large binary packages from pypi are difficult to install. They are likely to break due to unfulfilled dependencies or wrong dependency versions.
The problem is that the manylinux wheel standard cannot be fulfilled by many large packages, since they require non-python dependencies which are not available from pypi.

Fulfilling non-python dependencies for conda packages, on the the other hand, is easy, since those are fully declared and available from anaconda.

Independent from my infrastructure

Updating the databases for pypi releases requires constant crawling of meta data.
If I stopped running these crawlers, mach-nix users would be stuck with an outdated version of pypi metadata until someone decides to rehost the infrastructure (which is open sourced via nixops template).
For conda, mach-nix just requires repodata.json files. Those can be downloaded from anaconda anytime and do not require any extra infrastructure to maintain.

aarch64 support

It seems like some community maintained conda channels, like for example conda-forge, provide a lot more binary releases for aarch64 than pypi does.

Examples

Import mach-nix with conda support

let mach-nix = import (builtins.fetchGit {
  url = "https://github.com/DavHau/mach-nix";
  ref = "refs/heads/conda-beta";
}) {}; in
...

Build conda environment defined via environment.yml

... # import
mach-nix.mkPython {
  requirements = builtins.readFile ./environment.yml;
}

Select build variants and channels

... # import
mach-nix.mkPython {
  requirements = ''
    tensorflow >=2.3.0 mkl*
    blas * mkl*
    requests
  '';
  providers.requests = "conda-forge";
}

Include extra conda channels

Channels added via condaChannelsExtra are automatically appended to the default providers. This example uses impure fetching for simplicity. It’s better to manually download the repodata.json files and reference them locally.

let mach-nix = import (builtins.fetchGit {
  url = "https://github.com/DavHau/mach-nix";
  ref = "refs/heads/conda-beta";
}) {
  condaChannelsExtra.bioconda = [
    (builtins.fetchurl "https://conda.anaconda.org/bioconda/linux-64/repodata.json")
    (builtins.fetchurl "https://conda.anaconda.org/bioconda/noarch/repodata.json")
  ];
}; in
mach-nix.mkPython {
  requirements = ''
    kraken2
  '';
}
7 Likes

Released 3.1.0 (27 Nov 2020)

flakes lib, cli improvements, bugfixes

Features

  • expose the following functions via flakes lib:
    • mkPython / mkPythonShell / mkDockerImage / mkOverlay / mkNixpkgs / mkPythonOverrides
    • buildPythonPackage / buildPythonApplication
    • fetchPypiSdist / fetchPypiWheel
  • Properly manage and lock versions of nixpkgs and mach-nix for environments created via mach-nix env command.
  • Add example on how to use mach-nix with jupyterWith

Improvements

  • Improve portability of mach-nix env generated environments. Replace the platform specific compiled nix expression with a call to mach-nix itself, which is platform agnostic.
  • Mach-nix now produces the same result no matter if it is used through flakes or legacy interface. The legacy interface now loads its dependencies via flakes.lock.

Fixes

  • mkDockerImage produced corrupt images.
  • non-python packages passed via packagesExtra were not available during runtime. Now they are added to the PATH.
  • remove <nixpkgs> impurity in the dependency extractor used in buildPythonPackage.
4 Likes

Released 3.1.1 (27 Nov 2020)

fix cli

Fixes

  • Fix missing flake.lock error when using mach-nix cli.
3 Likes

Released 3.2.0 (11 Mar 2021)

bugfixes, ignoreCollisions

Features

  • add argument ignoreCollisions to all mk* functions

  • add passthru attribute expr to the result of mkPython, which is a string containing the internally generated nix expression.

  • add flake output sdist, to build pip compatible sdist distribution of mach-nix

Fixes

  • Sometimes wrong package versions were inherited when using the nixpkgs provider, leading to collision errors or unexpected package versions. Now, python depenencies of nixpkgs candidates are automatically replaced recursively.

  • When cross building, mach-nix attempted to generate the nix expression using the target platform’s python interpreter, resulting in failure

Package Fixes

  • cartopy: add missing build inputs (geos)

  • google-auth: add missing dependency six when provider is nixpkgs

5 Likes