Make a derivation for a Kicad plugin

Kikit is a plugin for Kicad that I would like to set up a development environment for with Nix. Even using normal virtualenv is somewhat of a challenge for this, because Kikit wants to talk to the Python bindings for Kicad which are not install-able via PyPi. It seems like Nix's ability to install an isolated, reproducible version of both Kicad and the Python dependencies for Kikit would make it a great problem for Nix to solve. But… I don’t know how to get it to work.

So far I have this:

{ pkgs ? import <nixpkgs> {} }:                          
                                                         
let kikit = pkgs.python3Packages.buildPythonApplication {
  pname = "KiKit";                                       
  src = ./.;                                             
  version = "0.4";                                       
  propagatedBuildInputs = [                              
    pkgs.kicad                                           
    pkgs.python3Packages.numpy                           
    pkgs.python3Packages.shapely                         
    pkgs.python3Packages.click                           
    pkgs.python3Packages.markdown2                       
    ];                                                   
  };                                                     
in                                                       
pkgs.mkShell {                                           
  buildInputs = [ pkgs.kicad kikit ];                    
}                                                        

This doesn’t build because python cannot find pcbnew which is the name of the binding which Kicad provides. I’ve tried to get this working all day, but haven’t found anything that was at a level I can understand. Can anyone point me in the right direction?

I will try to read it later (I will sleep now), but if it is a package request, what about to open an issue on the Github and track it here? Just fire it and link here.

1 Like

Thanks, I would appreciate you taking a look at it. It’s not quite a package request, (although I will have to open one I eventually I think). For now, I think what I need help with is how to make a nix-shell that has a python interpreter in it that can access the python bindings that come from KiCAD (which is not a python package).

Edit:
I’ve since come across this open issue, which seems relevant. Sounds like maybe the change needed is to the Kicad derivation so it supports this pattern?

i’ve had a go at this based on @FRidh 's work

since i’m not familiar with kikit, mind trying this?
it will rebuild kicad unless nixpkgs-unstable has caught up to FRidh’s bc3c3b4a59c

{ pkgs ? import (builtins.fetchTarball "https://github.com/evils/nixpkgs/tarball/f3ff36007c85ea201296c03fc55e457f7ff1ebdd") {} }
:

let kikit = pkgs.python3Packages.buildPythonApplication {
  pname = "KiKit";
  src = pkgs.fetchFromGitHub {
    owner = "yaqwsx";
    repo = "KiKit";
    rev = "c76c51e1dedeb79ad46414541d89c9a031133fef";
    sha256 = "1dmyvjm9bllgirli6jr12d2dvy9795qa49j0n2jhsq05hvp0p8gn";
  };
  version = "c76c51e";
  propagatedBuildInputs = [
    pkgs.python3Packages.numpy
    pkgs.python3Packages.shapely
    pkgs.python3Packages.click
    pkgs.python3Packages.markdown2
    pkgs.python3Packages.pybars3

    # adds kicad.base, kikit builds
    pkgs.kicad.py
    ];
  };
in
pkgs.mkShell {
  buildInputs = [
    kikit

    # wrapped kicad so it's user-usable
    pkgs.kicad
  ];
}

Wow, thank you for doing this! I was hoping for some pointers at best, but this is great! You even packaged PyBars and PyMeta! It did trigger a pretty big build for me that is still compiling (already been an hour and half or so). Once that finishes, I’ll test it and let you know for sure, but this really looks like it should work.

Presuming this does work, what are the next steps to get this upstream? I know of at least some other Kicad plugins (like this one) that are going to need the python bindings exposed like this to work with Nix, so from the perspective of a Kicad user it’s a pretty important feature of the package. Obviously, my Nix skills aren’t particularly strong yet but is there anything I can do to help? In the meantime I’ll be trying to learn from and understand the commits you made to get this working.

the missing python bindings have been gnawing at me since i saw #40835 11 days ago

i kinda had to package the dependencies of kikit because i couldn’t think of another addon to try
i’d probably not submit those without figuring out their checkPhases

if this works i’d open a PR with those changes in hopes of getting someone with python/python bindings know-how to review it
i’m in doubt about this working, i’m not familiar with kicad plugins, and this one doesn’t appear to be detected
i just tried running kicad from the shell defined in derivation i posted and it seems to be missing the wrapping (so it’s getting the bin/kicad from kicad.base aka kicad.py instead of the wrapped one from pkgs.kicad)

Well, I’m just glad there are more people than just me interested in figuring it out.

Sorry for the dumb question, but when you describe pkgs.kicad as “wrapped” what do you mean? In general, wouldn’t we want the “Kicad” that provides the python bindings to also provide the user Kicad binary?

You very likely have already done all this, but if you wanted to check the python bindings for yourself while I’m waiting on my build, you can test if you are able to import pcbnew. Kikit also has a CLI, so you should be able to run kikit --help, which I’m pretty sure fails if pcbnew isn’t available.

sorry for this slightly verbose answer, it is partially intended to document the kicad package

kicad.base provides the full build product of kicad
its binaries require some wrapping to work in nixos
this consists of a bash script that sets environment variables related to the GUI finding runtime dependencies
and importantly the kicad libraries (symbols/templates/footprints/packages3d)

kicad tries to discover its libraries either via an absolute path (not an option with nix)
relative to the binaries location (possible in nix, and was previously done) but this means a change to the libraries causes a rebuild of the binaries
or via an environment variable (such an option is missing for the i18n)

the output i refer to as pkgs.kicad (build result of …/kicad/default.nix) ties together the binaries from kicad.base and the libraries with a wrapper (and makes the GUI work)
and currently only exposes a /bin containing those wrapped binaries (and some CLI tools that didn’t require wrapping)

most kicad plugins probably require the bindings and FRidh pointed us to exposing the those
i’m not sure the functions accessed via those bindings are entirely functional as the binaries they have access to shouldn’t be able to find the libraries, and the other stuff in the wrapper

i suspect that if one wants to manipulate for example a 3D model via the bindings, the context provided by the wrapping will have to be relayed somehow, possibly by sourcing the wrapper in the environment running the tool that wants access to it

getting plugins to work will probably involve modifying the wrapper so kicad can find them
which, without knowing how kicad does that, i can’t do

Does one want to open a normal Python shell and use the bindings from there, or does one want to use a Kicad builtin Python shell and access packages such as kikit from there?

Thanks for your explanation, I have a much better understanding now. It’s unfortunate that Kicad is so challenging to package.

With respect to Kikit, I was probably misusing the term “plugin”. Kikit is a standalone program, and only calls into the bindings provided by Kicad so there is probably some use even if Kicad’s wrapped path cannot “see” Kikit. Also, since Kikit doesn’t manipulate the libraries at all, just board files, I’m maybe more optimistic that the binding will work OK. Still waiting on my build to finish up so I can test.

Update:
My build just finished and I’ve done a couple preliminary tests in @evils shell using nix-shell --pure. I’ve got to sign off for the day soon, but a couple things I’ve noticed in my first couple minutes of testing:

  1. Kikit seems to work just fine, and can generate panels from the CLI as I expect. There are other features of Kikit, but because I don’t use them yet I need to read up more to test them.
  2. I can open both Kicad and pcbnew with a few caveats.
  3. pcbnew seems to view a PCB fine, and it can find footprint libraries it seems. Is Kicad somehow finding the libraries I have installed outside the nix-shell? I’m running this on an Ubuntu machine which has Kicad installed.
  4. eeschema also seems to be able to find libraries. Again, not sure if this is an artifact of having Kicad installed outside the nix-shell
  5. the 3d viewer for both the board and the footprints doesn’t work.

I think there is use to calling into the bindings from a normal python shell. I want to be clear that I don’t speak for Kikit (I’m just a user), but part of it’s goal is to allow you to write a script which you independently run to generate a full fabrication package. At least currently, this always happens from outside Kicad.

kikit working is probably due to it being a CLI utility that just reuses some functions from kicad via the python bindings and nothing besides that and the provided files (at least, that’s what i assume it’s doing)

you have the unwrapped kicad binaries ${pkgs.kicad.base}/bin/ in your shell $PATH
for me, it’s not finding any new footprints but already used ones i believe are embedded in the pcb file

to get a completely clean setup, i think one needs to rm -r ~/.cache/kicad ~/.config/kicad and start a new project (or at least remove fp-info-cache from the project directory)

Yup, I agree with your assessment about how Kikit is working.

I’ve been trying to understand where this all leaves us. To make sure I’m tracking, I’ll restate the problem: In an ideal world we would be able to install Kicad in such a way that whenever it is run whether from the GUI, or when the python bindings are invoked the same wrapper environment is set. Currently, when the bindings are accessed there is no place to “hook in” and source the correct wrapper environment which is not ideal.

Does getting to the “ideal” scenario of the same Kicad enviornment from the GUI or from the bindings require upstream Kicad changes? (which sounds like a bigger challenge than just updating Nixpkgs) Is there a way to write a function that Nixpkgs can provide which can wrap utilities (like Kikit) which have a kicad.py binding dependency in the same enviornment that pkgs.kicad has? I obviously am not experienced enough to have a fully formed idea of how this would work, but I’m thinking something along the lines of extracting this part of the Kicad derivation into a function for programs with a Kicad dependency to use?

Hopefully this makes some sense and isn’t just rambling… Whatever I do is going to involve a fair amount of stumbling around in the dark (which I’m completely willing to do), but if you can steer me towards an approach that is more likely to work I’d appreciate it!

Even in this non-ideal state where the bindings don’t interact with the wrapped Kicad binaries in the way we would like, there is quite a bit of utility in exposing the python bindings. I know of other utilities designed to work with the Kicad python bindings that would probably work despite the wrapper issue. If nothing else, it would be nice to find a way to enable this level of functionality in upstream Nix.

i’ve opened a draft PR, #88923
in hopes of getting at least minimal bindings to work out of the box

as i’m not familiar with the mechanisms used by kicad plugins, how much context their python bindings require, or what doesn’t work with these changes, i can’t think of a good way forward

i suppose it’s possible to go back to the “one output for everything” where the libraries sit next to the source, then the binaries should be able to find those
however, then we would end up compiling kicad 4 times (vs 2 now, stable and unstable)
maybe something can be done with buildFHSUserEnv to still compose the final output but reuse the compiled source

a maybe useful half-measure could be to provide a blank wrapper to be reused for packaging plugins/utilities that require something from the wrapper

i’ve asked for help on the kicad forum at https://forum.kicad.info/t/23105

I’ve been following you pull request, and Kicad forum post. Is having a test for the KiCAD bindings the current blocker? I’m willing to help write some tests if you would like, but as you know a bit inexperienced with NIx/NixPkgs. Can you point me to a test elsewhere in NixPkgs that would be good for me to use as a template for this? How deep a “test” would we like? Would you suggest we start with each of the suggested plugins from your Kicad forum discussion?

I’ve also done a bit more research into the pcbnew bindings, which are currently the only ones exposed by Kicad. At first glance, the best docs I can find for the pcbnew bindings don’t appear to indicate any interaction with the Kicad libraries. So at least for those bindings, perhaps this is an indication that it’s “safe” to expose bindings that don’t have access to the same libraries as the wrapped Kicad binary, at least until upstream Kicad exposes a binding that needs them? I’m not sure about the plugins which are integrated into the Kicad gui, I don’t use any of those.

i think the next step is just to try out the suggested plugins and see what if anything goes wrong
i would probably end up packaging them
if that’s done, i’d write a nixos test that uses them (i’d start with the CLI only tools)

if i understand the replies on the kicad forum right, the pcbnew bindings are the only ones that exist
they also seem to indicate that the bindings don’t interact with the libraries, so it would probably be the plugins that would need to discover the libraries, and i’m not sure if any even try

i’m currently focused more on attempting to package wxPython 4.1.0 for an unrelated kicad issue
so if you can get even 1 more plugin to fail or succeed, that’d be great

Sounds good. I’ve been caught up in other work the past few days as well, but if I get a moment I’ll see if I can take a crack at trying out some more plugins and/or getting the check phases of the Kikit dependencies you already got working so they are more ready to be upstreamed. I’ll let you know here if I make progress.