Data Science on nixos nix, poetry, pip, mach-nix, pynixify: - all fail?

Help
6

Works with a recent nixpkgs, e.g. try

{ pkgs ? import (fetchTarball "https://github.com/NixOS/nixpkgs/tarball/nixos-unstable") { } }:
1 Like
7

you are right, it works.
(the thing is, that it would take a week to build a complex Data Science if it had to be compiled locally first)

  • any suggestion?
8

I just want to caution that a lot of machine learning frameworks are difficult to get working correctly even on a normal FHS installation. Doing so on nixos is another step of potential pain. (E.g. tensorflow is broken on nixpkgs constantly)

1 Like
9

that is the reason why I’m trying nixos… (besides broken tensorflow … )

  • conda doesn’t resolves the env any more
  • pip fails do to version constraints
  • …
10

welcome to python, when stuff works, it barely works, and when stuff is broken, it’s probably broken in many places.

The python packages on nixpkgs have had a lot of handwork go into them to make them “more coherent”. Unfortunately, the python ecosystem (with very few exceptions) is in a state of constant turbulence and breakages.

4 Likes
11

I maintain a consistent python environment for data scientists so they don’t have to. I use the buildEnv approach to create a derivation with all the packages they request which I then install in the default Nix profile (I use the single user Nix setup with a dedicated “admin” user). If a package in not available in nixpkgs I create a derivation for it using python-package-init. I also use overlays for tweaking parameters of the packages (MKL, GPU, etc.) and patching broken packages. Every 2-3 months I move the pinned nixpkgs to the latest unstable and rebuild the environment. Such an approach makes the versions consistent for all the teams/projects and the data scientists don’t have to learn nix.

8 Likes
12

This is probably the sanest thing to do

not familiar with it, what does it do?

13

There are a few of us in a slack channel sometimes exchanging tips.

https://join.slack.com/t/nix-data/shared_invite/enQtOTYyNjQzMDA0ODgzLTM3ZDE3Mjg3ZWI2N2ExOTNkOGYzMWU2NTEzZmIyNTE0OTg1NDZlZjYzM2JjNDA4ZTg5MTg3ZWE4NzM5NzI2NzQ

14

Is this the same group as in Workgroup:DataScience - NixOS Wiki
? Should I add the slack link there?

15

I built mach-nix assuming that python packages don’t have circular dependencies. Your examples have proven me wrong. This is now handled in version 2.2.1 together with another python 3.8 related bug.

Now your build should succeed.

16

Its the following tool written by @costrouc that generates a Nix expression from a PyPI package GitHub - nix-community/nixpkgs-pytools: Tools for removing the tedious nature of creating nixpkgs derivations [maintainer=@costrouc]

1 Like
17

Circular dependencies are currently quite problematic because Nix requires a DAG. To solve this issue, we should longer-term install a Python package in an output, but perform wrapping/patching to other Python packages in a second derivation (the one creating a Python environment).

2 Likes
18

That would be cool! I was thinking the same.
This would also improve caching effectivity.

Currently, if one runtime dependency changes, the package needs to be rebuilt. If we would add runtime deps only in a final wrapping, like you proposed, then cache invalidation is less likely to happen.
Mach-nix would benefit a lot from this, as one little change of a single package often leads to a mass rebuild of many other packages.

Reducing these “side effects” of runtime deps might allow to create a cache for mach-nix which caches a few different versions of the most common base dependencies and allow a faster build experience.

1 Like
19

Interesting, I’m making a similar tool which tries to do this for any package type: GitHub - jonringer/nix-template: Make creating nix expressions easy

20

tested pynixify via requirement file but (for now) it fails
build error

21

regarding mach-nix:

(on NixOS 20.03) why do I get Python 2.7 from home-manager
if I execute:

nix-shell shell_mach-nix__test.nix --show-trace
[nix-shell:] which python 
/home/user/.nix-profile/bin/python
# [nix-shell:} ls -lah /home/user/.nix-profile/bin/python
lrwxrwxrwx 1 root root 72 Jan  1  1970 /home/user/.nix-profile/bin/python -> /nix/store/5rzsb9q7v4qzbgwz3pp6kfzkf48kyzq8-home-manager-path/bin/python
[nix-shell:] python --version
Python 2.7.18

py27

22

python points to python2.7 for historical reasons. If you want to reference python3, then you must specify python3

23

good evening jonringer,

the link / config is

python = pkgs.python38;
24

then I’m not sure, :frowning:

25

The problem here was that the python derivation was loaded directly into nix shell instead of being wrapped in a mkShell.

I have to say I think it is a actually a really weird behaviour of nix-shell to not build the derivation by default. I know that nix-shell was original invented to debug builds and therefore not building the main derivation made sense.

But by now I assume that 95% of all uses of nix-shell (if not even more) are to load normal environments.

So why don’t we change the defualt behaviour of nix-shell to actually load derivations, while still allowing to use the build debug mode by passing --debug-build?

I’m very sure that this would have saved me a lot of time and confusion when I started using nix.

Isn’t it a bit strange that you always need to rewrite/extend your expression before you can use it in a shell?

3 Likes