Basic flake: run existing (python, bash) script

Hello,

I have a python script that depends on the selenium library and chromedriver; it constantly breaks due to system updates. I thought it might be a good candidate to employ nix and hopefully get something that works reliably over time. I also want to learn more about flakes.

I’ve spent the morning trying to figure out how to make the most basic and bare-bones flake possible to do this, hoping to avoid extra dependencies like flake-utils or poetry. I’m only interested in running on aarch64-darwin for now.

My goal is to be able to run nix run and have that result in python myscript.py being called.

Unfortunately, I can’t seem to figure out what the best approach for even this simple task is. It looks like I might need to use buildPythonApplication? NixOS - Nixpkgs 22.05 manual

Here is what I have so far, although I’ve now figured out that shellHook doesn’t do what I thought it would.

{
  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
  };

  outputs = { self, nixpkgs }:
    let
    system = "aarch64-darwin";
  pkgs = nixpkgs.legacyPackages.${system};
  python = pkgs.python3.withPackages (ps: with ps; [ selenium ]);
  in
  {
    packages.${system}.default =
      with import nixpkgs { inherit system; };
    pkgs.mkShell {
      nativeBuildInputs = [ python ];
      shellHook = ''
        ${python} ./myscript.py
        '';
    };
  };
}

I thought it might help me if anyone would be willing to contribute an example flake or two, ideally not depending on flake-utils just for instructive purposes, that upon nix run would:

  1. run a bash command: echo foo
  2. run an existing bash script, myscript.sh
  3. run an existing bash script that depends on e.g. ripgrep
  4. run a python command: print("foo")
  5. run an existing python script: ./myscript.py
  6. run an existing python script that depends on an arbitrary pypi package

My goal is mainly for adapting several existing single-purpose scripts. Obviously asking for 6 example flakes is a lot, but if anyone things they can pitch in a canonical, best-practices, dependency-free example of a flake that would work on a single system (aarch64-darwin in my case), I’d really appreciate it! I thought this would be fairly trivial, but after tinkering, googling, and reading all morning, I’d love to have some pointers.

Many thanks in advance.

EDIT: Changed unordered list to numbered to allow specifying which task something was directed at accomplishing.

Here’s an example for the first few, at least:

{
  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
  };

  outputs = { self, nixpkgs }:
  let
    system = "x86_64-linux";
    pkgs = nixpkgs.legacyPackages.${system};
    python = pkgs.python3.withPackages (ps: with ps; [ selenium ]);
  in
  {
    packages.${system} = {
      output1 = pkgs.writeScriptBin "myscript" ''
        echo foo
      '';

      output2 = pkgs.writeScriptBin "myscript" ./myscript.sh;

      output3 = pkgs.writeScriptBin "myscript" ''
        export PATH=${pkgs.lib.makeBinPath [ pkgs.hello ]}:$PATH
        ${./run-hello.sh}
      '';

      output4 =
        let
          tmpPyScript = pkgs.writeTextFile {
            name = "tmp-py-script";
            text = ''
              print("foo from python")
            '';
          };
        in
        pkgs.writeScriptBin "run-python" ''
          ${python}/bin/python ${tmpPyScript}
        '';
    };
  };
}

and these can be run with nix run .#output1 etc. in some directory.

e.g. as a gist at https://gist.github.com/rgoulter/35daa2c1152ac0b4381fd38c4ddb4069 with nix flakes, you can even run nix run git+https://gist.github.com/rgoulter/35daa2c1152ac0b4381fd38c4ddb4069#output1 which isn’t very helpful for this task, but seems a neat feature to keep in mind.

(IIRC, buildPythonApplication involves stuff like setup.py or whatever, which is more than I’m interested in).

For more intense Python stuff… at that point, you might want to take a look at https://github.com/DavHau/mach-nix or https://github.com/nix-community/poetry2nix

3 Likes

… Hmmm, lost my original reply unfortunately … oh well.

Thanks for your time!

It looks like adding default = self.packages.${system}.output1; lets you choose what is run with a bare nix run, which is great.

I found a few more helpful links while searching for lib.makeBinPath and a few other things that were new to me from your post:

I think searching specifically for "flake’ material may have excluded these from my earlier search attempts.

I also found a really great / thorough post here: Shell Scripts with Nix — ertt.ca

It looks like symlinkJoin may be a good way to accomplish #3, but for some reason it doesn’t seem to be actually setting the PATH:

#!/usr/bin/env bash
# foo.sh

echo "$SHELL"
command -v rg
echo foo | rg o
# flake.nix
{
  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
  };

  outputs = { self, nixpkgs }:
    let
      system = "aarch64-darwin";
      pkgs = nixpkgs.legacyPackages.${system};
      script = pkgs.writeScriptBin "say_foo" ''
        #!${pkgs.stdenv.shell}
        ${builtins.readFile ./foo.sh}
      '';
    in
    {
      packages.${system} = {
        default = pkgs.symlinkJoin {
          name = "say_foo";
          paths = [
            script
            pkgs.ripgrep
          ];
        };
      };
    };
}
$ nix run
/opt/homebrew/bin/bash
/opt/homebrew/bin/rg
foo

I thought nix flakes were “pure” and shouldn’t be grabbing these binaries from the parent environment’s PATH. Why is that happening? Shouldn’t pkgs.symlinkJoin be directing it to the nix-installed rg? Weird.

1 Like

Aha, writeShellApplication: nixpkgs/trivial-builders.nix at 588c4f214ee5c98d3925666d84555025e5e6ea5c · NixOS/nixpkgs · GitHub patches the path based on inputs.

Unfortunately, attempting it results in the below error right now:

error: Package ‘python3.10-pyopenssl-22.0.0’ in /nix/store/1njdlszpc3bchdcwk45c3ndb0nfmwqcr-source/pkgs/development/python-modules/pyopenssl/default.nix:73 is marked as broken, refusing to evaluate.

I’m not sure what is bringing in Python, it looks like it just runs shellcheck on the input, which I think is in Haskell, right?

Also, NIXPKGS_ALLOW_BROKEN=1 nix run --impure now downloads over a Gb of new stuff :man_facepalming:

Maybe I’d be better off with the manual writeScriptBin from @rgoulter above!

1 Like

Overriding the checkPhase works well, thought not sure why I’m still using my system bash.

#!/usr/bin/env bash
# foo.sh

echo "$SHELL"
command -v rg
echo foo | rg o
# flake.nix
{
  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
  };

  outputs = { self, nixpkgs }:
    let
      system = "aarch64-darwin";
      pkgs = nixpkgs.legacyPackages.${system};
    in
    {
      packages.${system}.default = pkgs.writeShellApplication {
        name = "say_foo";
        runtimeInputs = [ pkgs.ripgrep ];
        text = ''
          #!${pkgs.stdenv.shell}
          ${builtins.readFile ./foo.sh}
        '';
        checkPhase = "${pkgs.stdenv.shellDryRun} $target";
      };
    };
}
$ nix run
/opt/homebrew/bin/bash
/nix/store/c39pjjw72rcnli5fwbnhv529kwa6rksm-ripgrep-13.0.0/bin/rg
foo

EDIT: Nevermind; adding ps -p $$ to my foo.sh shows that it is indeed running in nix’s bash. Can also prove that by adding pkgs.ps to the runtimeInputs and running like so, where the path doesn’t even contain my system bash:

$ PATH=~/.nix-profile/bin nix run
38964 ttys037    0:00.00 /nix/store/l81df76j5jxr8lymk9zp9af94llkir94-bash-5.1-p16/bin/bash /nix/store/n5dnnq00ihhplg84hsll95ffnc3423i4-say_foo/bin/say_foo
/opt/homebrew/bin/bash
/nix/store/c39pjjw72rcnli5fwbnhv529kwa6rksm-ripgrep-13.0.0/bin/rg
foo

It is interesting that $SHELL is still set to my system bash. Even with:

$ env -i ~/.nix-profile/bin/nix run
  PID TTY           TIME CMD
39676 ttys037    0:00.00 /nix/store/l81df76j5jxr8lymk9zp9af94llkir94-bash-5.1-p16/bin/bash /nix/store/n5dnnq00ihhplg84hsll95ffnc3423i4-say_foo/bin/say_foo
/opt/homebrew/bin/bash
/nix/store/c39pjjw72rcnli5fwbnhv529kwa6rksm-ripgrep-13.0.0/bin/rg
foo

I guess that must be because it’s the shell used to spawn the nix process itself.

1 Like

writePython3Bin seems like the way to accomplish #4:

An example including dependencies:

{
  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
  };

  outputs = { self, nixpkgs }:
    let
      system = "aarch64-darwin";
      pkgs = nixpkgs.legacyPackages.${system};
      py3pkgs = pkgs.python3.pkgs;
    in
    {
      packages.${system}.default = pkgs.writers.writePython3Bin "say_foo" { libraries = [ py3pkgs.requests ]; } ''
        import requests
        status = requests.get("https://n8henrie.com").status_code
        print(status)
      '';
    };
}
$ NIXPKGS_ALLOW_BROKEN=1 nix run --impure
200

Accomplishing #5 and #6 seem pretty simple leveraging the same builtins.readFile as above:

# flake.nix
{
  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
  };

  outputs = { self, nixpkgs }:
    let
      system = "aarch64-darwin";
      pkgs = nixpkgs.legacyPackages.${system};
      py3pkgs = pkgs.python3.pkgs;
    in
    {
      packages.${system}.default = pkgs.writers.writePython3Bin "say_foo"
        {
          libraries = [ py3pkgs.requests ];
        } "${builtins.readFile ./foo.py}";
    };
}
# foo.py
import sys

import requests

print(sys.executable)
status = requests.get("https://n8henrie.com").status_code
print(status)
$ NIXPKGS_ALLOW_BROKEN=1 nix run --impure
/nix/store/gg1vps3aljdismx7rqps4m738gjcf9fp-python3-3.10.4-env/bin/python3.10
200

I guess the only part remaining is the “arbitrary PyPI dependency part.”

2 Likes

For number 6 (and 5 I think counts as well), I’ve spend several days reading and tinkering, and haven’t found a great solution.

mach-nix failed to install / work on my M1 mac. Other attempts at solving this problem (pypi2nix for example) are abandoned. It seems that the overall state of using Python with Nix is… not great. Which seems concerning, given that the popularity and utilization of Python dwarfs that of nix.

For the solution that more-or-less answers my initial question / concerns (no extra dependencies, nix run results in python myscript.py being called, using flakes):

# flake.nix
{
  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
  };

  outputs = { self, nixpkgs }:
    let
      system = "aarch64-darwin";
      pkgs = nixpkgs.legacyPackages.${system};
      py3pkgs = pkgs.python3.pkgs;
      venvDir = "./.venv";
      pypiDeps = [ "simplenet" ];
      shell = pkgs.mkShell
        {
          name = "python-env";
          buildInputs = [ py3pkgs.python py3pkgs.venvShellHook ];
          venvDir = ".venv";
          postVenvCreation = ''
            python3 -m pip install "${toString pypiDeps}"
          '';
        };

      runner = pkgs.writeShellApplication
        {
          name = "runme";
          checkPhase = ":";
          runtimeInputs = [ py3pkgs.python ];
          text = ''
            if [ -e "${venvDir}/bin/python" ]; then
              echo "Skipping venv creation, '${venvDir}' already exists"
            else
              rm -rf ./"${venvDir}"
              echo "Creating new venv environment in path: '${venvDir}'"
              ${py3pkgs.python.interpreter} -m venv "${venvDir}"
             "${venvDir}"/bin/python -m pip install --upgrade pip
             "${venvDir}"/bin/python -m pip install "${toString pypiDeps}"
            fi
            "${venvDir}"/bin/python < ./foo.py
          '';
        };
    in
    {
      apps.${system}.default = {
        type = "app";
        program = "${runner}/bin/runme";
      };
      devShells.${system}.default = shell;
    };
}
# foo.py
import sys

import simplenet

print(sys.executable)
print(simplenet.__version__)

With something like this, nix develop drops you into a dev environment within a virtualenv, and nix run runs foo.py with dependencies made available in the virtualenv. Both with create the virtualenv (only if it doesn’t seem to exist and have a bin/python that exists) and install dependencies in pypiDeps.

Unfortunately runner is more complex than I had hoped, as I don’t see a simple way to manually execute hooks (I get runHook: command not found if I try to add venvShellHook to runtimeInputs and then runHook venvShellHook, I don’t see any other way to run a hook with pkgs.writeShellApplication), which is what makes nix develop comparatively simple.

There was recently a thread with someone else discussing the same woes Why is it so hard to use a Python package?

Both with create the virtualenv (only if it doesn’t seem to exist and have a bin/python that exists) and install dependencies in pypiDeps .

For “I want to use the venv in the current directory”, I wonder if direnv / Python would be more suitable. Home · direnv/direnv Wiki · GitHub

Rather, I think having a nix run application make use of the current directory, and installing runtime dependencies with pip in order to do so, isn’t very idiomatic/‘clean’.

e.g. I updated my gist with your runner example, and

nix run git+https://gist.github.com/rgoulter/35daa2c1152ac0b4381fd38c4ddb4069#runner

for me runs

/nix/store/nznji0b6g1r2gl97bb6m27irqvppzn03-python3-3.10.4-env/bin/python3.10
v0.1.4

The Nix for the runner being:


      runner = pkgs.writeShellApplication
        {
          name = "runme";
          checkPhase = ":";
          runtimeInputs = [pythonWithSimplenet];
          text = ''
            "${pythonWithSimplenet}/bin/python" ${./foo.py}
          '';
        };

and pythonWithSimplenet being:

    simplenet = ps: ps.callPackage ./simplenet.nix { };
    pythonWithSimplenet = pkgs.python3.withPackages (ps: [ (simplenet ps) ]);

and the Nix description of simplenet as a Python package is in the flake. (With caveats that I skipped including dependencies for running the tests; and the numpy in nixpkgs that the flake points to is older than your simplenet required).

Yes, I saw that thread – and I agree with some of the sentiments expressed. For such an extremely popular and beginner-friendly language, I’m surprised that nix doesn’t seem to have an smooth process.

Perhaps – I’ve used direnv and similar tools in the past and didn’t care for them; I generally just have a config.env and use an alias like alias sv='source ./.venv/bin/activate; source config.env;'. But this also is adding a dependency, with one of the goals of this question being finding a solution that didn’t also add new dependencies, as I’m having a hard enough time troubleshooting nix at this point.

On that thread, I’ve been trying and trying to get mach-nix to work, as it seems to be the currently recommended approach in many similar conversations, and it’s gone nowhere. Installed as a flake, in nix-shell, and via pip, and on my Macbook I get odd errors when trying to generate a nix expression with virtually the easiest possible situation requirements.txt. With the flake + nix shell on my Macbook it doesn’t seem to install correctly (https://github.com/DavHau/mach-nix/issues/479). With a pip or nix-shell installation I get different errors entirely (https://github.com/DavHau/mach-nix/issues/482). On Linux it seems to work for old versions, but since I went to the currently recommended pyproject.toml project format, it can’t install simplenet>=0.1.3 even on linux (even though a wheel is available: https://files.pythonhosted.org/packages/8c/1b/d7b4ee5c49fc3213cebe53e55bd76bb2069b81e80696ba6cfa4959c16e94/simplenet-0.1.4-py3-none-any.whl). There doesn’t seem to be a way to specify a different revision of the pypi database from the CLI tool.

I would hope for a more obvious “right” way to do things for an incredibly popular language like python.

I agree, but it seems like a way to get things working for the moment as the alternative paths are… thorny, as far as I can tell.

Thank you again for your time and the example. Assuming that one doesn’t want to patch the dependencies (for example if I depended on a bugfix that was in the pinned version of numpy, as I haven’t run tests on the version you used), is the “right” way to just iteratively fetchPypi with each dependency, and all of its dependencies, get the hash once it gives you a hash mismatch error, and then do it again… until you’ve recursively gone through the entire dependency tree? I intentionally chose one of my projects with only a single dependency to simplify things, but I assume that one would need to walk the entire dependency tree… which seems like an exhausting process to do by hand!

I haven’t looked into the prospect too far, but one alternative that came to mind was:

$ pip download --dest vendor simplenet==0.1.4
...
$ ls -l vendor/
total 12476
-rw-r--r-- 1 n8henrie staff 12764532 Jul  1 08:35 numpy-1.22.4-cp39-cp39-macosx_11_0_arm64.whl
-rw-r--r-- 1 n8henrie staff     7437 Jul  1 08:35 simplenet-0.1.4-py3-none-any.whl

This seems to do a great job getting all of the dependencies in one fell swoop, afterwards it would seem relatively (?) easy to hash them and make them reproducible-ish inputs for a derivation (at least for a single machine, I don’t know about their metadata).

Perhaps – I’ve used direnv and similar tools in the past and didn’t care for them

Ah, fair that it may not be the tool to solve what you’re after.

I’ll say though that I like the combination of direnv+nix. (The nix integration means not needing to worry about a system-wide python, the python integration means not needing to worry about activating the virtual environment).

which seems like an exhausting process to do by hand!

Heh. Thus the motivation for tools like mach-nix or poetry2nix, which I see in the issue you’ve filed doesn’t support the way your package was packaged.

Without using a project like that, I guess it’s down to either “hope someone wrote the package in nixpkgs already”, or, yeah, come up with the expressions.

Just another small example to remind myself how (and that I need) to use pkgs.symlinkJoin and PATH manipulation with wrapProgram in order to use nix’s chromedriver instead of my system one.

# flake.nix
{
  inputs.nixpkgs.url = github:NixOS/nixpkgs/nixos-22.05;

  outputs = { self, nixpkgs }:
    let
      system = "aarch64-darwin";
      pkgs = nixpkgs.legacyPackages.${system};
      py3pkgs = pkgs.python3.pkgs;
      venvDir = "./.venv";

      pydeps = [ py3pkgs.python py3pkgs.selenium ];

      shell = pkgs.mkShell
        {
          name = "python-env";
          buildInputs = pydeps;
        };

      pyscript = pkgs.writers.writePython3Bin "tester"
        {
          libraries = pydeps;
        } "${builtins.readFile ./tester.py}";

      runner = pkgs.symlinkJoin
        {
          name = "run tester";
          paths = [ pyscript pkgs.chromedriver ];
          buildInputs = [ pkgs.makeBinaryWrapper ];
          postBuild = "wrapProgram $out/bin/tester --prefix PATH : $out/bin";
        };
    in
    {
      apps.${system}.default = {
        type = "app";
        program = "${runner}/bin/tester";
      };
      devShells.${system}.default = shell;
    };
}
# tester.py
import subprocess

import selenium

print(selenium.__version__, selenium.__file__, sep="\n")

subprocess.run(
    "which chromedriver; chromedriver --version;", shell=True
)
$ nix run
3.141.0
/nix/store/1hgvcb1fnw9gyz9sd7xplsf5d97cijrb-python3-3.9.13-env/lib/python3.9/site-packages/selenium/__init__.py
/nix/store/pc1qixi6pkyijrxmb9cqlc5di7zaqfnn-run-tester/bin/chromedriver
ChromeDriver 103.0.5060.24 (e47b049c438cd0a74dc95a011fceb27db18cb080-refs/branch-heads/5060@{#232})
1 Like
Hosted by Flying Circus.