First packaging attempt

Help
1

I am trying to create a nix package for Wyoming Satellite.

I am sure this would probably be a quick, easy work for some; but this is my first attempt to actually make a package.

The biggest issue I am running into is that the script/setup and script/run want the .venv to be in the source directory. They don’t seem to provide any way to override that.

I have tried going at this from a few different ways. My first inclination was to patch the scripts, but that seems very heavy handed when I actually should be able to provide a populated .venv directory as part of the build.

I then thought about trying to symlink the .venv directory to a writeable place like ${pwd}/_build/.venv, but that makes the whole thing impure.

I also ran into a problem with pyring-buffer not being a package available in python311Packages.

I am confused about how I am supposed to allow a derivation created by a fetch be mutable for the build steps before it gets marked read-only.

I would like to try to make this work so that the build allows script/setup to run its pip into the #{src}/.venv so that script/run can run using that.

The code I have is all over the place since I have been down so many paths, but I am pasting it here so that perhaps some of my misconceptions can be pointed out.

I keep trying to push myself away from just requesting this package. I would like to understand how to do it myself.

{
  description = "Wyoming Satellite";

  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-23.11";
    flake-utils.url = "github:numtide/flake-utils";
  };

  outputs = { self, nixpkgs, flake-utils }: 
  flake-utils.lib.eachSystem [ "x86_64-linux" "aarch64-linux" ] (system:
    let
      pkgs = nixpkgs.legacyPackages.${system};

      nativeBuildInputs = with pkgs; [
        (python3.withPackages(ps: with python311Packages; [ pip ]))
      ];

      src = pkgs.fetchFromGitHub {
        owner = "rhasspy";
        repo = "wyoming-satellite";
        rev = "HEAD";
        hash = "sha256-38nZ13dfxdS8h6MvoxRtMtYllyT+xYQR9dTU5hqZKQs=";
        # url = "github:rhasspy/wyoming-satellite";
      };

      posVenvCreation = ''
        unset SOURCE_DATE_EPOCH
        pip install -r requirements.txt
      '';

      shellHook = ''
        export PIP_PREFIX="$(pwd)/_build/pip_packages"
        cd ${src}
      '';
    in rec  {
      devShells.default = pkgs.mkShell {
        name = "wyoming-satellite";

        inherit nativeBuildInputs;
        inherit posVenvCreation;
        inherit shellHook;

      };
      packages.wyoming-satellite = pkgs.python311Packages.buildPythonPackage {
        name = "wyoming-satellite";

        inherit src;
        inherit nativeBuildInputs;

        buildPhase = ''
          ${src}/script/setup
        '';

      };
      packages.default = packages.wyoming-satellite;
    }
  );
}
2

I don’t think using a venv inside a derivation is going to work since Nix will block internet access while the derivation is building. You can go the venv-route in a devShell or similar, but that will not result in you getting a derivation. I don’t think you need to run the scripts at all, the package script does (which would be more appropriate than setup since the latter installs all dependencies) is to run "python3 -m setup.py bdist_wheel` which isn’t that special, so I assume if you get the dependencies right, the Nix Python builder should just build this package.

So instead of going the venv-route, you probably need to look at the dependencies in requirements.txt and package them if they are not already packaged for Nix, then create a package for wyoming-satellite. You will likely need to adapt the requirements.txt since they seem to be pinning the versions very tightly down to the patch version. Unlikely that you will find that exact combination in nixpkgs.

Overall, this doesn’t seem like a very friendly package to start out with.

2 Likes
3

Thank you for the clarification.

I tried including the packages using the pkgs.python3.withPackages (ps: [ … ]) and there were two problems.

The first problem is that pyring-buffer doesn’t yet appear to be available in nixpkgs. I suppose that means I should try to package that one first. The second problem is that script/run specifically calls out ${src}/.venv and won’t run if it isn’t available. It doesn’t seem to have an override available for the path or a way to just use the current virtualEnv.

The reason I chose this package is because I would like to use this package on a raspberry pi and I would like to build that raspberry pi using nix. Do you think I should just request the package port at nixos.org or do you think this is something I will be able to figure out?

4

At the same time, could it really be that packaging pyring-buffer was this easy?

        pyring-buffer = pkgs.python311Packages.buildPythonPackage {
          name = "pyring-buffer";

          src = pkgs.fetchFromGitHub {
            owner = "rhasspy";
            repo = "pyring-buffer";
            rev = "HEAD";
            hash = "sha256-bHhcBU4tjFAyZ3/GjaP/hDXz2N73mCChTNYHsZyBCSM=";
            # url = "github:rhasspy/pyring-buffer";
          };

        };
2 Likes
5

I now see what you mean about ignoring the use of pip.
As a test I used my devShell, now refactored to:

      devShells.default = pkgs.mkShell {
        name = "wyoming-satellite";

        nativeBuildInputs = with pkgs; [
          (python3.withPackages(ps: with python311Packages; [ 
            pip 
            wyoming 
            zeroconf 
            packages.pyring-buffer 
          ]))
        ];

        src = pkgs.fetchFromGitHub {
          owner = "rhasspy";
          repo = "wyoming-satellite";
          rev = "HEAD";
          hash = "sha256-38nZ13dfxdS8h6MvoxRtMtYllyT+xYQR9dTU5hqZKQs=";
          # url = "github:rhasspy/wyoming-satellite";
        };

        shellHook = ''
          # export PIP_PREFIX="$(pwd)/_build/pip_packages"
          cd ${devShells.default.src}
        '';

      };

And with my new pyring-buffer, it does “appear” to contain the right dependencies.
Theoretically at this point I think I should be able to just run python3 -m wyoming_satellite from that directory, but I get this error:

Traceback (most recent call last):
  File "<frozen runpy>", line 189, in _run_module_as_main
  File "<frozen runpy>", line 148, in _get_module_details
  File "<frozen runpy>", line 112, in _get_module_details
  File "/nix/store/khv9a7lqdpam5aw133zxhf1abxaamw38-source/wyoming_satellite/__init__.py", line 2, in <module>
    from .satellite import (
  File "/nix/store/khv9a7lqdpam5aw133zxhf1abxaamw38-source/wyoming_satellite/satellite.py", line 14, in <module>
    from wyoming.error import Error
ModuleNotFoundError: No module named 'wyoming.error'

I can verify from python3 that I can import wyoming. I don’t understand why I don’t have this wyoming.error class.

6

Looks like you are on the right path. I checked the wyoming repo and the Error module seems to be a fairly recent addition. You are pinning Nixpkgs 23.11 which only has 1.2.0 whereas unstable has 1.4.0. Maybe that already fixes your issue? They seem to pin 1.4.2, so maybe you can do the update yourself and even submit a PR to Nixpkgs.

At the same time, could it really be that packaging pyring-buffer was this easy?

Packing for Nix is actually not hard if the packages adhere to best practices.

1 Like
7

Thank you for looking at that for me. It is a lot easier to continue just knowing I am on the right path. Did I read somewhere that if I post a request for a package on nixos.org and the only difference is the version, that a bot somewhere would do the update? Would that be the better way to have the version of wyoming updated?

I tried my flake with nixos-unstable. It does get past the Error problem, but it is missing a different class at that point.

I would like to submit my pyring-buffer (since that works) into a PR for nixpkgs, but I am not sure exactly what the procedure is for that. I can guess some of it, but I guess I won’t really know what parts I am missing until I try.

Right now I have pyring-buffer pinned to HEAD. I am assuming submitting a package without pinning to a tag is probably frowned on, but pyring-buffer does not appear to have any releases or TAGS. I suppose that means I just choose the current revision?

8

I made the update to a local branch of nixpkgs to use wyoming-1.4.2 and now my wyoming_satellite flake runs.
Github won’t let me push the branch with the version update. How are PR’s submitted? (I apologize if this information is easily available).

9

You fork the repo, push to a branch in your fork, and create a Pull Request from Github. For pyring-buffer there is a version 1.0.0 on PyPi. You can switch to fetchPyPi to use that, there are plenty of packages in Nixpkgs as examples to follow. Cool that you got it working!

1 Like
10

Please read https://github.com/NixOS/nixpkgs/blob/005862f60d11a2949b696191d67315b8ee69d8fa/CONTRIBUTING.md ; also second using fetchPyPi. Congrats and keep nixing!

2 Likes
11 
          pname = "pyring_buffer";
          version = "1.0.0";

          src = pkgs.fetchPypi {
            inherit pname version;
            format = "wheel";
            python = "py3";
          };

This does not seem to be able to find the package at: https://files.pythonhosted.org/packages/py2.py3/p/pyring_buffer/pyring_buffer-1.0.0-py3-none-any.whl
I am not sure how I am supposed to find these pypi details, but this is the name of the file when you try to download manually at pypi.org.

I wonder if I should start a new topic for this?

12

try pyring-buffer? with the hyphen; edit - nah, hmmm. I wonder what’s up with that? Anyway, if fetchFromGitHub works I think that’s fine - see below

1 Like
13

Oh, it’s a binary release so we can only build from source so fetching from GitHub is the correct way. I hate to link to the unofficial user wiki but scroll down to the bit about 404 and fetchPypi: Packaging/Python - NixOS Wiki @genson when you do open a PR feel free to add me as a reviewer :slightly_smiling_face: I’ll probably follow your example to officially add a program I packaged for work that might be useful in the cache

1 Like
14

Not very exciting, but its my first:
python3Packages.wyoming: 1.4.0 → 1.4.2

I plan to follow up with pyring-buffer and wyoming-satellite.

2 Likes
15

wyoming-satellite: init at 1.0.0 by mweinelt · Pull Request #277407 · NixOS/nixpkgs · GitHub already takes care of the packaging. It is currently waiting on mental capacity to migrate the module from its predecessor homeassistant-satellite.

2 Likes
16

Thank you for creating this package. It took me some time to fiddle around with the “extraArgs”.
I also had to enable the ports on the firewall.

I ended up with:

  services.wyoming.satellite = {
    enable = true;
    name = "nixos satellite";
    user = "system";
    uri = "tcp://0.0.0.0:10700";
    sounds.awake = builtins.fetchurl "https://github.com/rhasspy/wyoming-satellite/raw/master/sounds/awake.wav";
    sounds.done = builtins.fetchurl "https://github.com/rhasspy/wyoming-satellite/raw/master/sounds/done.wav";
    extraArgs = [ 
      "--debug" 
      "--wake-word-name=ok_nabu" 
      "--wake-uri=tcp://127.0.0.1:10400" 
    ];
  };
  services.wyoming.openwakeword = {
    enable = true;
    preloadModels = [
      "ok_nabu"
    ];
    uri = "tcp://0.0.0:10400";
    extraArgs = [ "--debug" ];
  };