Help with python flake build from local source

Help
1

I’ve come up with the following flake.nix for building/developing my keepmenu project but I can’t get it to work with hatch-vcs to set the current git version.

{
  description = "Dmenu/Rofi/Wofi frontend for Keepass databases";

  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs";
  };

  outputs = {
    self,
    nixpkgs,
  }: let
    systems = ["x86_64-linux" "i686-linux" "aarch64-linux"];
    forAllSystems = f:
      nixpkgs.lib.genAttrs systems (system:
        f rec {
          pkgs = nixpkgs.legacyPackages.${system};
          commonPackages = builtins.attrValues {
            inherit
              (pkgs.python3Packages)
              python
              pykeepass
              pynput
              ;
          };
        });
  in {
    devShells = forAllSystems ({
      pkgs,
      commonPackages,
    }: {
      default = pkgs.mkShell {
        packages = commonPackages;
      };
    });
    packages = forAllSystems ({
      pkgs,
      commonPackages,
    }: {
      default = pkgs.python3Packages.buildPythonApplication {
        name = "keepmenu";
        pname = "keepmenu";
        format = "pyproject";
        src = ./.;
        nativeBuildInputs = builtins.attrValues {
            inherit
              (pkgs.python3Packages)
              hatchling
              hatch-vcs
              ;
            };
        propagatedBuildInputs = commonPackages;
        meta = {
          description = "Dmenu/Rofi/Wofi frontend for Keepass databases";
          homepage = "https://github.com/firecat53/keepmenu";
          license = pkgs.lib.licenses.gpl3;
          maintainers = ["firecat53"];
          platforms = systems;
        };
      };
    });
  };
}

If I specify pulling the source from pypi or github with the version and hash it seems to work fine. Otherwise, I get the following errors when running nix build:

Sourcing python-remove-tests-dir-hook
Sourcing python-catch-conflicts-hook.sh
Sourcing python-remove-bin-bytecode-hook.sh
Sourcing pip-build-hook
Using pipBuildPhase
Using pipShellHook
Sourcing pip-install-hook
Using pipInstallPhase
Sourcing python-imports-check-hook.sh
Using pythonImportsCheckPhase
Sourcing python-namespaces-hook
Sourcing python-catch-conflicts-hook.sh
@nix { "action": "setPhase", "phase": "unpackPhase" }
unpacking sources
unpacking source archive /nix/store/70kwddhg6dsyhwsaa42z107137qr06h0-mbig43nslrlq32a51wxxnvwmd4kjb43a-source
source root is mbig43nslrlq32a51wxxnvwmd4kjb43a-source
setting SOURCE_DATE_EPOCH to timestamp 315619200 of file mbig43nslrlq32a51wxxnvwmd4kjb43a-source/tests/tests.py
@nix { "action": "setPhase", "phase": "patchPhase" }
patching sources
@nix { "action": "setPhase", "phase": "updateAutotoolsGnuConfigScriptsPhase" }
updateAutotoolsGnuConfigScriptsPhase
@nix { "action": "setPhase", "phase": "configurePhase" }
configuring
no configure script, doing nothing
@nix { "action": "setPhase", "phase": "buildPhase" }
building
Executing pipBuildPhase
Creating a wheel...
WARNING: The directory '/homeless-shelter/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag.
Processing /build/mbig43nslrlq32a51wxxnvwmd4kjb43a-source
  Running command Preparing metadata (pyproject.toml)
  Traceback (most recent call last):
    File "/nix/store/66qrgkmha6dmcac29nvxbjsbrh77jlin-python3.10-pip-23.0.1/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
      main()
    File "/nix/store/66qrgkmha6dmcac29nvxbjsbrh77jlin-python3.10-pip-23.0.1/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/nix/store/66qrgkmha6dmcac29nvxbjsbrh77jlin-python3.10-pip-23.0.1/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 152, in prepare_metadata_for_build_wheel
      whl_basename = backend.build_wheel(metadata_directory, config_settings)
    File "/nix/store/x1nnn2lapxqkk6c7m5f0n863wx03l6b2-python3.10-hatchling-1.13.0/lib/python3.10/site-packages/hatchling/build.py", line 56, in build_wheel
      return os.path.basename(next(builder.build(wheel_directory, ['standard'])))
    File "/nix/store/x1nnn2lapxqkk6c7m5f0n863wx03l6b2-python3.10-hatchling-1.13.0/lib/python3.10/site-packages/hatchling/builders/plugin/interface.py", line 93, in build
      self.metadata.validate_fields()
    File "/nix/store/x1nnn2lapxqkk6c7m5f0n863wx03l6b2-python3.10-hatchling-1.13.0/lib/python3.10/site-packages/hatchling/metadata/core.py", line 243, in validate_fields
      _ = self.version
    File "/nix/store/x1nnn2lapxqkk6c7m5f0n863wx03l6b2-python3.10-hatchling-1.13.0/lib/python3.10/site-packages/hatchling/metadata/core.py", line 128, in version
      self._version = self._get_version()
    File "/nix/store/x1nnn2lapxqkk6c7m5f0n863wx03l6b2-python3.10-hatchling-1.13.0/lib/python3.10/site-packages/hatchling/metadata/core.py", line 226, in _get_version
      version = self.hatch.version.cached
    File "/nix/store/x1nnn2lapxqkk6c7m5f0n863wx03l6b2-python3.10-hatchling-1.13.0/lib/python3.10/site-packages/hatchling/metadata/core.py", line 1412, in cached
      raise type(e)(message) from None
  LookupError: Error getting the version from source `vcs`: setuptools-scm was unable to detect version for /build/mbig43nslrlq32a51wxxnvwmd4kjb43a-source.

  Make sure you're either building from a fully intact git repository or PyPI tarballs. Most other sources (such as GitHub's tarballs, a git checkout without the .git folder) don't contain the necessary metadata and will not work.

  For example, if you're using pip, instead of https://github.com/user/proj/archive/master.zip use git+https://github.com/user/proj.git#egg=proj
  error: subprocess-exited-with-error
  
  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /nix/store/1r6n7v2wam7gkr18gxccpg7p5ywgw551-python3-3.10.12/bin/python3.10 /nix/store/66qrgkmha6dmcac29nvxbjsbrh77jlin-python3.10-pip-23.0.1/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py prepare_metadata_for_build_wheel /build/tmpu79c1gzw
  cwd: /build/mbig43nslrlq32a51wxxnvwmd4kjb43a-source
  Preparing metadata (pyproject.toml) ... e[?25le[?25herror
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

Is there any way to get this working without either having to keep the flake up-to-date with the current version or having to switch to using poetry? Also, I’m open to any other feedback on the flake itself, as I’m still new to Nix.

Thanks!

2

Just FYI, /homeless-shelter is what nix sets $HOME to when building a derivation to prevent builds from accessing to the user’s home directory. So this is an expected warning for pipBuildPhase.

I’m not quite sure what you mean. I tried to reproduce this with the following steps:

$ git clone git@github.com:firecat53/keepmenu.git 
$ cd keepmenu
$ vim flake.nix # Copied your flake in as posted
$ git add --intent-to-add flake.nix
$ nix build
warning: Git tree '/mnt/s/repos/keepmenu' is dirt
$ result/bin/keepmenu -h
usage: keepmenu [-h] [-a AUTOTYPE] [-d DATABASE] [-k KEYFILE] [-t]

Dmenu (or compatible launcher) frontend for Keepass databases
[...]

But as you can see, it just builds without an issue.

Is there some special branch that you’re working on?

3

I was working on the develop branch. It should have the same flake.nix as above. Would you be able to checkout the develop branch and try? I think the biggest difference is that it uses a pyproject.toml instead of setup.py. That’s probably the cause, but is it fixable?

Thanks!

4

I guess the problem is that a flake only contains the tracked files when copied to the store. You can see that here:

$ nix flake metadata 
warning: Git tree '/mnt/s/repos/keepmenu' is dirty
Resolved URL:  git+file:///mnt/s/repos/keepmenu
Locked URL:    git+file:///mnt/s/repos/keepmenu
Description:   Dmenu/Rofi/Wofi frontend for Keepass databases
Path:          /nix/store/7h033mvhxgvicq51vvrw304vr2j25byi-source
Last modified: 2023-06-21 04:19:16
Inputs:
└───nixpkgs: github:NixOS/nixpkgs/842e90934a352f517d23963df3ec0474612e483c
$ ls -lavhF /nix/store/7h033mvhxgvicq51vvrw304vr2j25byi-source
total 80K
dr-xr-xr-x 1 root root    282 Jan  1  1970 ./
drwxrwxr-t 1 root nixbld  13M Jul  8 16:46 ../
-r-xr-xr-x 1 root root     96 Jan  1  1970 .gitignore*
-r-xr-xr-x 1 root root    35K Jan  1  1970 LICENSE*
-r-xr-xr-x 1 root root    422 Jan  1  1970 Makefile*
-r-xr-xr-x 1 root root   2.9K Jan  1  1970 README.md*
-r-xr-xr-x 1 root root   2.9K Jan  1  1970 config.ini.example*
dr-xr-xr-x 1 root root     60 Jan  1  1970 docs/
-r-xr-xr-x 1 root root    508 Jan  1  1970 flake.lock*
-r-xr-xr-x 1 root root   1.5K Jan  1  1970 flake.nix*
dr-xr-xr-x 1 root root    266 Jan  1  1970 keepmenu/
-r-xr-xr-x 1 root root   3.9K Jan  1  1970 keepmenu.1*
-r-xr-xr-x 1 root root   4.6K Jan  1  1970 keepmenu.1.md*
-r-xr-xr-x 1 root root   1.6K Jan  1  1970 pyproject.toml*
-r-xr-xr-x 1 root root     24 Jan  1  1970 requirements.txt*
dr-xr-xr-x 1 root root     72 Jan  1  1970 tests/

The .git directory is missing, so this part of your pyproject.toml causes the issue:

[tool.hatch.version]
source = "vcs"

As Hatch can’t find the current version. I’m not sure how to best work around this limitation. Hatch has to get the version from a different source.

The easiest way is to just set fallback-version in the hatch version table:

[tool.hatch.version]
source = "vcs"
fallback-version = "1.3.2"

Though this seems a bit sketchy to me.

Another way to do this would be to add a version variable to your derivation function:

default = pkgs.python3Packages.buildPythonApplication {
        name = "keepmenu";
        pname = "keepmenu";
        version = "1.3.2";
        format = "pyproject";
        src = ./.;

As all input attributes to a derivation exist as environment variables during build time, you can then let Hatch read this as an environment variable:

[tool.hatch.version]
source = "env"
variable = "version"

This already works, but maybe you’re not 100% happy with this, as you’d have to bump the version manually?

I assume this also means that building without nix is less convenient now as you have to set the version environment variable explicitly before. Not sure that’s a downside for you.

A semi-automatic solution would be using self.rev and self.lastModified to generate a unique version number.

    systems = ["x86_64-linux" "i686-linux" "aarch64-linux"];
    baseVersion = "1.3.1";
    version = if (self ? rev) then "${baseVersion}-${self.rev}" else "${baseVersion}-${builtins.toString self.lastModified}";
    forAllSystems = f:
[...]
        pname = "keepmenu-${version}";
        inherit version;
        format = "pyproject";

This will result in a version like 1.3.1-4fde66ee247c0d3af9ca68b430254e1f7cf2bd09 for clean trees and like 1.3.1-1688833425 for dirty trees. You can also replace rev with shortRev to get something more manageable like this: 1.3.1-06abf0e. For full releases, you could manually set the version to what you want.

There was a PR merged just two weeks ago that will also add dirtyRev and dirtyShortRev, so you don’t need lastModified anymore, but that will only available int the next release of Nix.

5

Wow, thanks so much for the detailed answer! It’s a lot to think about.

I feel like the primary reason I added the flake.nix was as an easy way to get a development shell with the dependencies included. The ability to nix build and nix run is a nice-to-have, but not really the primary goal.

I suspect a majority of keepmenu users are not using Nix and so I need to keep the typical install methods functional (pip install --user, etc.).

I’ll need to think about the options you provided and play with them a bit. Thanks again for the feedback!!

6

Yes, that makes sense. There’s actually another option that I forgot to mention; you can use a path Flake instead of a git Flake. This means the whole directory is copied to the store before building, and so hatch-vcs will work:

nix build path:.

You’ll have to add git to your nativeBuildInputs, otherwise it won’t be available during build time, but that’s it.

The disadvantage of this is that users will not be able to install keepmenu easily as a flake anymore by referring to it as github:firecat53/keepmenu, as path Flakes can only be local paths. But for making it easy for contributors to test whether their changes work, this could be a good workaround.

7

That does work. Some more questions (thanks so much for your patience!):

  1. Can the path:. argument to nix build be made default just for that flake? That way you can just run nix build or nix run instead of having to add the path:. argument.

  2. I’m struggling with the development workflow in this setup. I understand that nix can’t be used similar to a pip install -e . to be able to make a code change and then immediately run the app because the source is copied to the nix store. I added pip to the mkShell command. So with this workflow:

$ nix develop
$ pip install -e . --prefix $TMPDIR/  (note: all the keepmenu dependencies seem to be in place already in the nix store)
$ $TMPDIR/bin/keepmenu
Traceback (most recent call last):
File "/tmp/nix-shell.JeCure/bin/keepmenu", line 5, in <module>
  from keepmenu.__main__ import main
ModuleNotFoundError: No module named 'keepmenu'

However, this works:

❯ python
Python 3.10.12 (main, Jun  6 2023, 22:43:10) [GCC 12.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from keepmenu.__main__ import main
>>> main()

The installed app at $TMPDIR/bin/keepmenu seems to be using the same python as it is when called from the python repl.

❯ cat $TMPDIR/bin/keepmenu
#!/nix/store/1r6n7v2wam7gkr18gxccpg7p5ywgw551-python3-3.10.12/bin/python3.10
# -*- coding: utf-8 -*-
import re
import sys
from keepmenu.__main__ import main
if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(main())
❯ which python
/nix/store/1r6n7v2wam7gkr18gxccpg7p5ywgw551-python3-3.10.12/bin/python

I have found plenty of posts around setting up the development environment, but I can’t find anyone showing how to actually install and iteratively develop their apps! There’s direnv, devshell, nix-direnv, etc etc, but I haven’t really seen how those would help, other than making some of the boilerplate switching between environments easier.

Just a simple hatch shell seems pretty straightforward compared to the nix flow in this case :slight_smile: Well aware that I’m still climbing the learning curve though.

Thanks!

8

Yes, you can’t build easily directly inside the working directory, but nix can do something pretty close:

How about nix shell? :wink:

I’ll try to demonstrate what it does:

$ which python
/usr/bin/python
$ nix develop
$ which python
/nix/store/1r6n7v2wam7gkr18gxccpg7p5ywgw551-python3-3.10.12/bin/python
$ which keepmenu
keepmenu not found
$ nix shell path:.
$ which keepmenu
/nix/store/mh6mlwfw7y0m8cl59rzgr8k64w448if7-keepmenu/bin/keepmenu

So basically, nix develop puts you in a shell where all the dependencies are present that are needed for building your default package (most probably packages.x86_64-linux.default). You don’t even need the devShells output for this! Try it, if you remove devShells, you can still run nix develop and it will still work!
Basically, it puts you in the same environment that the scripts run by nix build will be in by default. specifying devShells will override that behaviour so you can add additional stuff that is needed for development but not building, for example.

And nix shell builds the package and then puts you in a shell where the default package is available.In this specific case (PATH diff abridged):

$ env > env-no-nix
$ nix shell path:.
$ diff env-no-nix <(env)
71c71
< SHLVL=1
---
> SHLVL=2
92c92
< PATH=/home/felix/.nix-profile/bin:/nix/var/nix/profiles/default/bin:/home/felix/.nix-profile/bin
---
> PATH=/home/felix/.nix-profile/bin:/nix/store/xd0ykfrrlqwi0wgb6lc64hshyn99cf1k-keepmenu/bin:/nix/var/nix/profiles/default/bin:/home/felix/.nix-profile/bin
106c106
< OLDPWD=/home/felix
---
> OLDPWD=/home/felix/repos/keepmenu

There’s actually some discussion about potentially renaming nix shell because this distinction is somewhat confusing, though I personally don’t agree with that.

The reason is probably that this wildly differs between projects. nix shell is nice and all, but it requires rebuilding the whole package from scratch every time, and so Nix itself has dedicated documentation about how to build and re-build and how to run tests individually inside a development shell instead of running nix shell every time.

But this brings us to your specific problem again:

So the issue here is that pip expects that it can just add stuff to your import path and that this will be picked up by the next python invocation, but Nix tries hard to avoid side-effects like that, and does so successfully. I couldn’t find out what pip actually does to achieve this, but the bottom line is, it doesn’t work, we have to manually add the repository’s directory to the PYTHONPATH so python actually finds it:

$ nix develop
$ pip install -e . --prefix $TMPDIR/
$ export PYTHONPATH=$PYTHONPATH:$PWD
$ python $TMPDIR/bin/keepmenu

And voilá, now it runs as expected, using exactly the dependencies locked by your flake.

Though I’m curious how often you’d want to do that when you can just

$ nix develop
$ python -m keepmenu

instead.

When running python in interactive mode, the current working directory is always searched for modules in addition to PYTHONPATH, basically replacing the step we added to the workflow above.

Yeah I agree. For me direnv is just a convenience feature so I don’t have to type nix develop manually, which is pretty cool, but not a game-changer.

No, unfortunately. I tried overriding the self input, but that only gets evaluated after Nix already determined what type of flake it’s working with, and there’s no builtin option for this.

If you want to make things more convenient, you could use a command-runner like just, and add that to the development shell so people only have to run nix develop once and just build, not worrying about the details underneath.

You could also use this to automate the workflow above. just run, just install or just test have a nice ring to them.

Thank you for all your work! I’ve used keepmenu and other tools of yours in the past and am thrilled to give something back :smiling_face_with_three_hearts:

9

Wow, awesome reply, thanks so much!! I’m glad you’ve found some use for the tools I’ve made to scratch my own itch :slight_smile:

The tip about just running python -m keepmenu was perfect. I actually just added an alias to devShells:

    devShells = forAllSystems ({
      pkgs,
      commonPackages,
    }: {
      default = pkgs.mkShell {
        packages = commonPackages;
        shellHook = ''
          alias keepmenu="python -m keepmenu"
        '';
      };
    });

Which totally solves the problem of being able to edit and run, edit and run using nix develop path:. without needing a pip install -e anywhere. So now I can run any of nix run path:., nix develop path:. and nix shell path:. and get expected behavior for each one. Need to document this still in the keepmenu docs.

I’ll have to use this for a bit for some actual development work to make sure it works as it seems to. Guess that means sitting down and clearing some PRs and issues!

Thanks again for all the support!

1 Like