Package Python Script w/ Dependency from GitHub

Help
1

Description

I wrote a python script that converts the bookmarks.html file exported by Firefox into Nix syntax that’s accepted by Home Manager’s programs.firefox.profiles.<name>.bookmarks. The script only has one dependency, but it’s not published to PyPi (or anywhere else other than its repo). I’m attempting to include this as a dependancy with the goal of packaging the script for other Nix users.

What I’ve Tried

This is the dependency needed and bellow is my shell.nix.

with import <nixpkgs> {};
with pkgs.python3Packages;

buildPythonPackage rec {
	pname = "html2nix";
	version = "0.0.1";
	src = ./.;

	format = "pyproject";

	propagatedBuildInputs = [
		setuptools
		fetchFromGitHub {
			owner = "FlyingWolFox";
			repo = "Netscape-Bookmarks-File-Parser";
			rev = "v1.1";
			hash = "112816d44bf4c0da8e8f442aed370020e16594e8888c8ddb10a699779dc666eb";
		}
	];
}

When I run nix-shell from the same directory as the script and pyproject.toml I get:

error:
       … while evaluating the attribute 'drvPath'

         at /nix/store/irsmypz8l2qv6xdpqpi9pxv0hjv5afyx-nixos-23.11/nixos/lib/customisation.nix:249:7:

          248|     in commonAttrs // {
          249|       drvPath = assert condition; drv.drvPath;
             |       ^
          250|       outPath = assert condition; drv.outPath;

       … while calling the 'derivationStrict' builtin

         at /builtin/derivation.nix:9:12: (source not available)

       (stack trace truncated; use '--show-trace' to show the full trace)

       error: Dependency is not of a valid type: element 2 of propagatedBuildInputs for python3.11-html2nix-0.0.1

In addition to fetchFromGitHub I’ve also tried fetchurl and fetchgit with identical results.

Finally, I’ve tried replacing fetchFromGitHub with:

  • git+github.com:FlyingWolFox/Netscape-Bookmarks-File-Parser.git
  • git+github.com:FlyingWolFox/Netscape-Bookmarks-File-Parser
  • github.com:FlyingWolFox/Netscape-Bookmarks-File-Parser.git
  • "git@github.com:FlyingWolFox/Netscape-Bookmarks-File-Parser.git"
  • "git@github.com:FlyingWolFox/Netscape-Bookmarks-File-Parser.git"
  • etc

I always get build input <theAbove> does not exist.

Help!

I’m new to the process of packaging as well as Nix. Any assistance would be appreciated.

System: x86 NixOS 23.11 python 3.11

Side note: I was inspired to dive into packaging after seeing this post.

2

I imagine there are 2 things here:

  1. This is just a fetcher; it’s just going to fetch source from this repo at this rev. It won’t be a built executable or library or whatnot. If the package you’re fetching is also a python package, you’d likely use buildPythonPackage on it as well.
  2. You may need to wrap this whole bit in parens.
1 Like
3

This is the entire shell.nix file:

with import <nixpkgs> {};
with pkgs.python3Packages;

buildPythonPackage rec {
	pname = "html2nix";
	version = "0.0.1";
	src = ./.;

	format = "pyproject";

	propagatedBuildInputs = [
		setuptools
		fetchFromGitHub {
			owner = "FlyingWolFox";
			repo = "Netscape-Bookmarks-File-Parser";
			rev = "v1.1";
			hash = "112816d44bf4c0da8e8f442aed370020e16594e8888c8ddb10a699779dc666eb";
		}
	];
}

Is this what you meant by wrap it in buildPythonPackage?

4

No–my comment only pertained to the part I quoted. Netscape-Bookmarks-File-Parser is presumably a python package? You’re just including its source in propagatedBuildInputs. It isn’t getting built as a python package.

5

Oh, I see. So I’ll need to do the same thing I’m doing for my package for NetscapeBookmarksFileParser because it is just source.

Thanks!

1 Like
6

Did you test if bookmarks folder are working? Currently I’m noting that, if you deploy bookmark folders, they are not shown in Firefox.

7

I’ve tested that the bookmarks.html exported by Firefox can be converted to Nix syntax via my script and that Firefox does pick up all bookmarks (and folders) when that syntax is passed to Home Manger’s programs.firefox.profiles.<name>.bookmarks.

I’ve noticed that the Netscape Bookmarks standard… doesn’t exist. So each browser exports different data with a slightly different layout. If you manually created the value assigned to programs.firefox.profiles.<name>.bookmakrs then that might cause issues (just a guess).

tl;dr: When I programmatically convert Firefox exports into nix, it works. :person_shrugging:

8

So after more messing about last night, and sleeping on it, I’m thinking the best route might be to package the dependency (NetscapeBookmarksFileParser) myself and publish it to PyPi. I think doing so would make the process of importing it into my package trivial. It might also give me practice packaging Python.

Does this sound like the best approach? Also, does anyone know of any nix packages that are written in Python that I could look at as a reference? I’ve looked through this page, but there’s still a lot I’m confused about.

9

You don’t need to put it up on pypi to build it as a python package. You just need something along the lines of (untested…):

with import <nixpkgs> {};
with pkgs.python3Packages;

buildPythonPackage rec {
	pname = "html2nix";
	version = "0.0.1";
	src = ./.;

	format = "pyproject";

	propagatedBuildInputs = [
		setuptools
		(buildPythonPackage {
			pname = "TODO";
			version = "TODO";
			src = fetchFromGitHub {
				owner = "FlyingWolFox";
				repo = "Netscape-Bookmarks-File-Parser";
				rev = "v1.1";
				hash = "112816d44bf4c0da8e8f442aed370020e16594e8888c8ddb10a699779dc666eb";
			};
			# ... could have its own dependencies, though it doesn't look like this one does
		})
	];
}
1 Like
10

Let’s goooooooo. We have progress! It attempts to download the source now, however, it now triggers a hash mismatch. However, I don’t think it’s a mismatch of the git hash because the error talks about hash mismatch in fixed-output derivation... (full error text at bottom).

Just in case I’ll go through the process I went throught to generate the hash. I downloaded the tar.gz v1.1 release from here, ran shasum -a 256 Netscape-Bookmarks-File-Parser-1.1.tar.gz, and got 112816d44bf4c0da8e8f442aed370020e16594e8888c8ddb10a699779dc666eb. I then updated what you (@abathur) provided as seen bellow:

with import <nixpkgs> {};
with pkgs.python3Packages;

buildPythonPackage rec {
	pname = "html2nix";
	version = "0.0.1";
	src = ./.;

	format = "pyproject";

	propagatedBuildInputs = [
		setuptools
		(buildPythonPackage rec {
			pname = "NetscapeBookmarksFileParser";
			version = "1.1";
			src = fetchFromGitHub {
				owner = "FlyingWolFox";
				repo = "Netscape-Bookmarks-File-Parser";
				rev = "v${version}";
				sha256 = "112816d44bf4c0da8e8f442aed370020e16594e8888c8ddb10a699779dc666eb";
			};
		})
	];
}

The hash mismatch error:

these 2 derivations will be built:
  /nix/store/07rfg466ynqh7a9vb5gdbimg7cxksbjg-source.drv
  /nix/store/wxlg6dr1d3fh9114605hnpk41kp5xzzq-python3.11-NetscapeBookmarksFileParser-1.1.drv
building '/nix/store/07rfg466ynqh7a9vb5gdbimg7cxksbjg-source.drv'...

trying https://github.com/FlyingWolFox/Netscape-Bookmarks-File-Parser/archive/v1.1.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 18987    0 18987    0     0  13929      0 --:--:--  0:00:01 --:--:-- 42571
unpacking source archive /build/v1.1.tar.gz
error: hash mismatch in fixed-output derivation '/nix/store/07rfg466ynqh7a9vb5gdbimg7cxksbjg-source.drv':
         specified: sha256-ESgW1Ev0wNqOj0Qq7TcAIOFllOiIjI3bEKaZd53GZus=
            got:    sha256-zCdwfOkYvGPEdWYxo4Tas1JDK60zjyxWzfuflP4y38U=
error: 1 dependencies of derivation '/nix/store/wxlg6dr1d3fh9114605hnpk41kp5xzzq-python3.11-NetscapeBookmarksFileParser-1.1.drv' failed to build

BTW, thanks for all your help @abathur!

11

I assumed the hash in your example was right, but it sounds like it isn’t. You should just be able to replace the sha256 line with:

hash = "sha256-zCdwfOkYvGPEdWYxo4Tas1JDK60zjyxWzfuflP4y38U=";
1 Like