Maximize code reuse between nixpkgs-compatible expression and overlay

Help
1

If a package I want to use is missing from nixpkgs, the options to get around this include:

  1. Create an overlay which adds the package locally.
  2. Add the package to a local clone of nxipkgs (… and submit a PR for its inclusion).

I would like to understand how to make the two as compatible as possible, or how to write code that is as reusable-in-both-contexts as possible.

For example, here is a contribution to nixpkgs that was accepted a while ago:

{ buildPythonPackage
, fetchPypi
, lib
, pytestCheckHook
}:

buildPythonPackage rec {
  pname = "pytest-instafail";
  version = "0.4.2";

  src = fetchPypi {
    inherit pname version;
    sha256 = "10lpr6mjcinabqynj6v85bvb1xmapnhqmg50nys1r6hg7zgky9qr";
  };

  checkInputs = [ pytestCheckHook ];
  pythonImportsCheck = [ "pytest_instafail" ];
  meta = {
    description = "pytest plugin that shows failures and errors instantly instead of waiting until the end of test session";
    homepage = "https://github.com/pytest-dev/pytest-instafail";
    license = lib.licenses.bsd3;
    maintainers = [ lib.maintainers.jacg ];
  };
}

What is the minimal modification to this which would allow it to be used as a local overlay?

Most obviously, the nixpkgs version (BTW, what is a short and correct name for ‘the expression which defines a/the package in nixpgs’ ?) receives everything it uses as an arbitrary set of arguments (in this case: buildPythonPackage, fetchPypi, lib, pytestCheckHook), while overlays are restricted to exactly self: super:. So it seems that I would need to prepend each occurrence of the nixpkgs-style parameters in the body of the expression, with self. or super., in order to make it usable in an overlay, which is a right royal PITA, whose cost grows with the complexity of the expression.

More generally, I find that there’s a tension between how things should look inside nixpkgs and elsewhere, and I find code reuse in different contexts in Nix to be quite challenging.

2

You do not have to do that, in your overlay, you can do something like:

self: super: {
  mypackage = super.callPackage pkgs/mypackage {};
}

where pkgs/mypackage is the directory containing the default.nix for the package. And callPackage will handle the arguments.

For Python packages you can use python3.pkgs.callPackage (replacing python3 by the versioned Python attribute that you want to use).

Edit: if you want to overlay the Python package set itself, see: Using python.withPackages with nixpkgs-overlays does not work · Issue #26487 · NixOS/nixpkgs · GitHub

3

Damn! I actually knew this, as I’ve done it a few times in my home-manager, but somehow my brain got lost and didn’t connect.

Thank you.

4

On a related note, I’m trying to restructure a monolithic shell.nix which started off thus

{ py ? "38"
, nixpkgs-commit-id ? "8e78c2cfbae71720869c35b6710d652bf28b37cc"
}:
let
  nixpkgs-url = "https://github.com/nixos/nixpkgs/archive/${nixpkgs-commit-id}.tar.gz";
  pkgs = import (builtins.fetchTarball { url = nixpkgs-url; }) {};
  python = builtins.getAttr ("python" + py) pkgs;
  pypkgs = python.pkgs;
  ... etc.

the idea being to allow choosing a specific Python version with nix-shell --argstr py 37.

I’m trying to separate the concerns of:

  • creating a nixpkgs-compatible expression for the project
  • pinning nixpkgs
  • enabling a development environment via nix-shell

so the new shell.nix should look something like this

# { py ? "38" }:

let
  # python = builtins.getAttr ("python" + py) pkgs;
  # pypkgs = python.pkgs;
  nixpkgs = import ./nix/nixpkgs.nix;
  pkgs = import nixpkgs {
    config = {};
    overlays = [
      (import ./nix/overlay.nix)
    ];
  };

in pkgs.my-derivation-added-via-overlay

except that I don’t see how to deal with the python-version selection (the commented-out lines).

5

If you want to use the selected version in the overlay, you can make overlay.nix contain a function that takes py argument and returns an overlay, instead of just overlay. Then you will be able to import it as import ./nix/overlay.nix { inherit py; }.

Of course, this will no longer allow you to use the overlay.nix as an overlay directly so if you want that, you will need real-overlay.nix containing import ./overlay.nix { py = "something"; }.

6

This is a standard structure I used in the past: https://www.reddit.com/r/NixOS/comments/8tkllx/standard_project_structure/

These days flakes are filling in a similar role in providing a standard and I’d recommend using them as they are better in many ways. Flakes - NixOS Wiki

7

Which is essentially what I was doing, but I was unsatisfied with it precisely because, as you point out, it’s not usable as a genuine overlay any more.

Yes, I see … I’ll have to think through it carefully to see how it fits in with everything else, when I next have some time to spend on Nix. Thanks for the suggestion!

8

Yes, I had come across it in the past, and my attempts were, indeed, influenced by it.

Sigh. Flakes.

Yes, I’ve been wanting to use them for some time, but I’ve been holding off because their unstable status and my limited time available for all things Nix, didn’t seem like a good fit.

But maybe it’s time to take the plunge.