Add python apps to flake project built with crate2nix

Help
1

I have a flake which uses crate2nix to build a Rust project which provides a number of binaries. The project also provides some utilities written in Python, that are regularly used by the developers, but which are also of interest to the users. This is a scientific research project, so

  • stuff initially written in Python may well be a prototype that gets rewritten in Rust later on,
  • the distinction between developer and user is somewhat nebulous.

The project could reap significant benefit from allowing users not at all involved in development to simply use both

  • nix run <my-flake>#<thing-that-happens-to-be-written-in-Rust-today>
  • nix run <my-flake>#<thing-that-happens-to-be-written-in-Python-today>

and not care or even notice which language is used to implement the utility, rather than mostly building and executing the code in the development environment, using lower-level tools which rub the implementation language in your face.

Can you suggest how to add python-implemented apps to this project? Will I have to forgo the use of crate2nix in order to do so?

2

Is there any reason this doesn’t work out of the box? Can’t you take the derivations created by crate2nix and put them in packages, and then replace them with different derivations depending on python when you swap them out?

As long as the name used in the packages attrset is the same, nix should happily take the new (python based) derivation and build that instead.

I have little experience with crate2nix, so forgive me if you’re using it to generate a flake.nix somehow - that would seem like an antipattern to me :slight_smile:

3

If I had multiple rust packages and multiple python packages then, yes, I think it would be trivial. But I only have one package: the root crate of the project generated by crate2nix, which contains all the Rust binaries in the project. My toy model for experimenting with these things is here.

My problem is that cargo2nix is a black box (to me at least) which takes my Cargo.toml and turns it into a Nix derivation. I don’t know how to tell cargo about the python stuff that’s knocking about, consequently I don’t know how to inject python stuff into this single package.

IIUC, packages.<system>.<name> corresponds to nix build <flake>#<name>, and consequently nix build <my-flake> creates something with the structure

result
└── bin
   ├── prog1
   ├── prog2
   ├── prog3
   └── progN

which contains all my executables, which correspond to

src/bin
├── prog1.rs
├── prog2.rs
├── prog3.rs
└── progN.rs

in a single package. I like this because I don’t want to burden the users with having to do anything more complicated that nix build <flake> or nix shell <flake>.

There is a 1-to-1 correspondence between src/bin/<X>.rs and result/bin/<X>. In these terms, the question becomes: is there any way of injecting some python stuff into result/bin without forgoing the services of crate2nix?

1 Like
4

A quick hack I can think of is to create a handful of wrapping packages that all depend on the original derivation created by cargo2nix, and copy out the individual binaries without actually building anything. If there are inter-dependencies, or more than just binaries, derivations that do nothing but wrapProgram to pick out the individual binaries as scripts is more suitable, at the cost of indirection and requiring all packages to be in the executing store even if just one is used (which isn’t too bad with small, static rust binaries?).

There may be a way to split apart an existing derivation, which would be a bit less ugly, but I’m not aware of such a thing.

There may be more satisfactory ways if you don’t treat cargo2nix as a complete black box, though I don’t see one myself. If you’re willing to do some upstream work I’m sure there would be interest in exposing the --package flag from cargo for this somehow.

1 Like
5

If I understand correctly: you have a flake, which has a derivation using crate2nix which has a bunch of rust binaries. You’d like the flake to provide some way to have nix build my-flake#some-output so that ./result/bin/ has the rust binaries, as well as Python scripts (from the same repository). Or even nix run my-flake#some-program (which may or may not be from crate2nix).

I think “how to tell crate2nix about Python stuff” is the wrong perspective.

For having nix build produce a result which has the binaries from crate2nix and python binaries, buildEnv sounds like what you want. Nixpkgs Reference Manual

e.g. with something like:

commit c76d56eec33450a0d52776af0595e42795d633f7
Author: Richard Goulter <blah@example.com>
Date:   Tue Feb 15 21:53:42 2022 +0700

    python

diff --git a/flake.nix b/flake.nix
index a7ca43a..33797d9 100644
--- a/flake.nix
+++ b/flake.nix
@@ -94,6 +94,19 @@
         in
         rec {
           packages.${name} = project.rootCrate.build;
+          packages."${name}-python" = pkgs.python3Packages.buildPythonApplication {
+            pname = "${name}-python";
+            version = "1.0";
+
+            src = ./.;
+          };
+          packages."${name}-env" = pkgs.buildEnv {
+            name = "${name}-env";
+            paths = [
+              packages.${name}
+              packages."${name}-python"
+            ];
+          };
 
           # ========== nix build =========================================================
           defaultPackage = packages.${name};
@@ -111,6 +124,9 @@
           apps.hdf = utils.lib.mkApp { drv = packages.${name}; name = "hdf"; };
           apps.ogl = utils.lib.mkApp { drv = packages.${name}; name = "ogl"; };
 
+          apps.three = utils.lib.mkApp { drv = packages."${name}-python"; name = "three.py"; };
+          apps.four  = utils.lib.mkApp { drv = packages."${name}-python"; name = "four.py";  };
+
           # ========== nix develop ========================================================
           devShell = pkgs.mkShell {
             inputsFrom = builtins.attrValues self.packages.${system};
diff --git a/setup.py b/setup.py
new file mode 100644
index 0000000..cad3454
--- /dev/null
+++ b/setup.py
@@ -0,0 +1,8 @@
+#!/usr/bin/env python
+
+from distutils.core import setup
+
+setup(name='foobar',
+      version='1.0',
+      scripts=['src/bin/three.py', 'src/bin/four.py'],
+     )
diff --git a/src/bin/four.py b/src/bin/four.py
new file mode 100644
index 0000000..4f16e47
--- /dev/null
+++ b/src/bin/four.py
@@ -0,0 +1,2 @@
+#!/usr/bin/env python3
+print("this is the FOURTH executable")
diff --git a/src/bin/three.py b/src/bin/three.py
new file mode 100644
index 0000000..b06f9ae
--- /dev/null
+++ b/src/bin/three.py
@@ -0,0 +1,2 @@
+#!/usr/bin/env python3
+print("this is the THIRD executable")

you get nix run .#three for Python, and nix build .#<name>-env to get ./result/bin with the rust and python programs.

1 Like
6

Thank you so much. buildEnv is the key concept that was completely missing from my (meagre) Nix knowledge. And the working code sample undoubtedly saved me many hours of frustration and failure.

1 Like
7

If the Python scripts that have been added in this way need to import a non-standard Python module which is provided by Nix, how should these modules be made available to the scripts?

1 Like
8

You’d use the propagatedBuildInputs attribute in the argument to buildPythonApplication. The NixOS Wiki Python page Python - NixOS Wiki and Nixpkgs 23.11 manual | Nix & NixOS are your friends.

Writing the Nix expression for packages in Pypi which aren’t in nixpkgs is typically straightforward, I think.

Another option, though I’m not well-versed in Python enough to comment: poetry2nix looks interesting. Developing Python with Poetry & Poetry2nix: Reproducible flexible Python environments This blogpost makes an argument for why it should be preferred over the Nix-only approach.

1 Like